Cloud-based application of visual effects to video

Abstract

A server system (300) receives from a first client device (100) a video stream (190, 190') for the video conferencing session, and receives from the first client device visual effects information for one or more visual effects to apply to the video stream, the server system applies one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams (192, 194), and transmits the one or more modified video streams to one or more other client devices participating in the video conferencing session.

Description

Technical Field

[0001] Claim of Priority This application claims priority based on U.S. Patent Application No. 17 / 738,176 filed on May 6, 2022, and the entire content thereof is incorporated herein by reference in its entirety.

[0002] This disclosure relates to video processing for video telephony (which may also be referred to as video calls or video conferences) conducted over a network. More specifically, the embodiments disclosed herein relate to video processing operations for video conferences that are executed in the cloud by one or more servers rather than by a client device.

Background Art

[0003] Video telephony has a remote communication system that transmits audio and visual signals using a computing device, thereby enabling multiple people to participate in a conversation in real time from different locations.

[0004] Some video telephony applications support various visual effects such as background blur and brightness adjustment. However, not all client devices (e.g., laptops, tablets, smartphones, etc.) have sufficient performance to render these visual effects without degrading the performance of the client device and / or the video conferencing application.

Summary of the Invention

[0005] Aspects and advantages of embodiments of this disclosure are partially shown in the following description, or can be learned from the description, or can be learned through the practice of exemplary embodiments.

[0006] In one or more exemplary embodiments, a server system (e.g., a video processing server) comprises one or more memories configured to store instructions, and one or more processors configured to execute instructions stored in the one or more memories, wherein the one or more processors execute the instructions to receive a video stream relating to a video conferencing session from a first client device, receive visual effects information relating to one or more visual effects to be applied to the video stream from the first client device, apply one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams, and transmit one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0007] In some embodiments, one or more processors in the server system are configured to send one or more modified video streams to a first client device.

[0008] In some embodiments, one or more processors of the server system are configured to send a notification to the first client device indicating that, in response to the media quality of the corrected video stream sent to the first client device being below a threshold level and / or the latency of the corrected video stream sent to the first client device being above a threshold level, the first client device will be responsible for controlling the application of one or more visual effects to the video stream, and / or to stop sending one or more corrected video streams to one or more other client devices participating in the video conferencing session.

[0009] In some embodiments, one or more processors in the server system are configured to generate multiple modified video streams, each having a different resolution, based on received visual effects information, and to send one of the multiple modified video streams to each of the one or more other client devices participating in the video conferencing session, based on the data transfer rate between the server system and each client device.

[0010] In some embodiments, the visual effects information includes information about one or more settings for one or more visual effects to be applied to the video stream. In some embodiments, one or more settings include a blur radius for a background blur effect to be applied to at least a portion of the video stream. In some embodiments, one or more settings include a brightness level for a light and dark effect to be applied to at least a portion of the video stream.

[0011] In some embodiments, the video stream transmitted from the first client device is the user's self-view on the first client device. In some embodiments, in response to the user's self-view being changed to a size exceeding a threshold level, one or more processors of the server system are configured to: stop applying one or more visual effects to the video stream; send a notification to the first client device indicating that the first client device is responsible for applying one or more visual effects to the video stream; and / or stop transmitting one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0012] In one or more exemplary embodiments, a computing device (e.g., a client device) comprises one or more memories configured to store instructions, and one or more processors configured to execute instructions stored in the one or more memories, wherein the one or more processors execute the instructions to determine whether to process video relating to a video conferencing session in a first mode or a second mode; in response to a decision to process the video in a first mode, generate a first video stream, send the first video stream to a server system, and send visual effects information relating to one or more visual effects to be applied to the first video stream to the server system; and in response to a decision to process the video in a second mode, generate a second video stream by applying one or more visual effects to the first video stream, and send the second video stream to the server system or another server system.

[0013] In some embodiments, one or more processors in a computing device are configured to receive from a server system a modified video stream in which one or more visual effects are applied to a first video stream based on visual effects information sent to the server system.

[0014] In some embodiments, one or more processors in a computing device are configured to determine whether to process the video in a second mode and / or stop transmitting the first video stream to the server system in response to the media quality of the corrected video stream being below a threshold level and / or the latency of the corrected video stream received from the server system being above a threshold level.

[0015] In some embodiments, the visual effects information includes information about one or more settings for visual effects to be applied to a first video stream. In some embodiments, one or more settings include a blur radius for a background blur effect to be applied to at least a portion of the first video stream. In some embodiments, one or more settings include a luminance level for a brightness effect to be applied to at least a portion of the first video stream.

[0016] In some embodiments, the video includes a self-view of a user of a computing device, and one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on the size of the self-view. In some embodiments, when the size of the self-view is below a threshold level, one or more processors of the computing device are configured to determine whether to process the video in a first mode, and when the size of the self-view is above a threshold level, one or more processors of the computing device are configured to determine whether to process the video in a second mode.

[0017] In some embodiments, the video includes multiple self-views of a user of a computing device, one or more processors of the computing device are configured to identify the self-view having the largest size from among the multiple self-views, and one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on the size of the self-view having the largest size.

[0018] In some embodiments, the visual effects information includes instructions for the server system not to send the first video stream to other client devices participating in the video conferencing session if one or more visual effects cannot be applied to the first video stream.

[0019] In one or more exemplary embodiments, a computer implementation for a computing device (e.g., a client device) includes: determining whether to process video relating to a video conferencing session in a first mode or in a second mode; in response to a decision to process the video in the first mode, generating a first video stream, transmitting the first video stream to a server system, transmitting visual effects information relating to one or more visual effects to be applied to the first video stream to the server system; and in response to a decision to process the video in the second mode, generating a second video stream by applying one or more visual effects to the first video stream, and transmitting the second video stream to the server system or another server system.

[0020] In one or more exemplary embodiments, a computer implementation for a server system (e.g., a video processing server) includes: receiving a video stream relating to a video conferencing session from a first client device; receiving visual effects information relating to one or more visual effects to be applied to the video stream from the first client device; applying one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams; and transmitting one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0021] In one or more exemplary embodiments, a computer-readable medium (e.g., non-temporary computer-readable medium) is provided that stores instructions executable by one or more processors of a client device and / or a server system. In some embodiments, the computer-readable medium may store instructions that cause one or more processors to perform one or more operations of any of the methods described herein (e.g., operations of a server system and / or operations of a client device). The computer-readable medium may also store additional instructions for performing other aspects of the server system and client device, as well as corresponding methods of operation, as described herein.

[0022] These features, aspects, and advantages of various embodiments of this disclosure, as well as other features, aspects, and advantages, will be better understood by referring to the following description, drawings, and appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, serve to illustrate the relevant principles.

[0023] A detailed description of exemplary embodiments intended for those skilled in the art is provided herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0024] [Figure 1] This disclosure illustrates an exemplary video conferencing system according to one or more exemplary embodiments. [Figure 2A] An exemplary flowchart for a video processing server according to one or more exemplary embodiments of the present disclosure is shown. [Figure 2B] An exemplary flowchart for a video processing server according to one or more exemplary embodiments of the present disclosure is shown. [Figure 3] The following are exemplary block diagrams of a client device and a video processing server according to one or more exemplary embodiments of the present disclosure. [Figure 4]An exemplary display of a video conferencing session on a client device, according to one or more exemplary embodiments of the present disclosure, is shown. [Figure 5] An exemplary display showing a visual effect applied to a user's self-view during a video conferencing session on a client device is shown. [Figure 6] A flowchart of an exemplary and non-limiting computer-implemented method, according to one or more exemplary embodiments of the present disclosure, is shown. [Figure 7] A flowchart of an exemplary and non-limiting computer-implemented method, according to one or more exemplary embodiments of the present disclosure, is shown. [Figure 8] A flowchart of an exemplary and non-limiting computer-implemented method, according to one or more exemplary embodiments of the present disclosure, is shown. [Figure 9] A flowchart of an exemplary and non-limiting computer-implemented method, according to one or more exemplary embodiments of the present disclosure, is shown. [Figure 10] A flowchart of an exemplary and non-limiting computer-implemented method, according to one or more exemplary embodiments of the present disclosure, is shown.

Mode for Carrying Out the Invention

[0025] Hereinafter, embodiments of the present disclosure in which one or more examples are shown in the drawings are referred to, and like reference numerals in the drawings indicate like elements. Each example is provided as an explanation of the present disclosure and is not intended to limit the present disclosure. In fact, it will be apparent to those skilled in the art that various changes and modifications may be made to the present disclosure without departing from the scope or spirit of the present disclosure. For example, features illustrated or described as part of one embodiment can be used with another embodiment to create yet another embodiment. Therefore, the present disclosure is intended to cover modifications and variations that fall within the scope of the appended claims and their equivalents.

[0026] The terms used herein are for illustrative purposes only and are not intended to limit and / or restrict the disclosure. The singular forms “a,” “an,” and “the” are intended to include the plural unless the context clearly indicates otherwise. In this disclosure, terms such as “including,” “having,” and “comprising” are used to specify features, numbers, steps, actions, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more features, numbers, steps, actions, elements, components, or combinations thereof.

[0027] In this specification, terms such as first, second, third, etc. may be used to describe various elements, but it should be understood that the elements are not limited to these terms. Rather, these terms are used to distinguish one element from another. For example, without departing from the scope of this disclosure, the first element may be referred to as the second element, and the second element may be referred to as the first element.

[0028] When one element is referred to as being "connected" to another, it will be understood that the expression includes examples of direct connection or joining, as well as connections or joining with one or more other elements intervening between those elements.

[0029] The term "and / or" includes a combination of multiple related items listed, or any one of the multiple related items listed. For example, the expression or phrase "A and / or B" includes item "A", item "B", and the combination of items "A and B".

[0030] Furthermore, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B and at least one of C.

[0031] According to an exemplary embodiment, the video processing for video conferencing may be performed on the cloud by one or more servers rather than by client devices, thereby freeing client devices from processing tasks (e.g., central processing unit / graphics processing unit (CPU / GPU) tasks) that could cause overheating, degraded display quality, or other performance degradation of the client devices. According to an exemplary embodiment, one or more servers receive a video stream relating to a video conferencing session from the client devices. One or more servers also receive visual effects information relating to one or more visual effects to be applied to the video stream from the client devices. Based on the received visual effects information, one or more servers apply one or more visual effects to the video stream to generate one or more modified video streams and send one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0032] According to embodiments of this disclosure, a video stream is acquired from a client device, and media features corresponding to visual effects to be added to the video stream are processed in the cloud by one or more servers, which are hereinafter referred to as video processing servers. A video processing server is a server (or combination of servers) configured to receive a video stream from a client device (possibly via an intermediate server) and to transmit the video stream (possibly in a modified form) to one or more other client devices. In exemplary embodiments, a video processing server may modify a video stream by adding one or more visual effects to the video stream based on information received from a client device about one or more visual effects to be applied to the video stream. A video processing server may also change the resolution of a modified or unmodified video stream based on the requirements of each client device receiving a modified or unmodified video stream. Thus, a video processing server has a video stream as input from a client device and multiple outputs corresponding to multiple modified or unmodified video streams with different resolutions according to the requirements of each receiving client device. In an exemplary embodiment, when the video processing server modifies the video stream, the modified video stream is sent not only to other client devices participating in the video conference, but also to the client device that sent the video stream to the cloud. Therefore, users of the client devices can also view the modified video stream with visual effects applied to it during the video conference.

[0033] According to an exemplary embodiment, a client device decides whether to process the video relating to a video conferencing session in a first mode or a second mode. In response to deciding to process the video in the first mode, the client device generates a first video stream, sends the first video stream to the server, and sends visual effects information to the server regarding the visual effects to be applied to the first video stream. In response to deciding to process the video in the second mode, the client device generates a second video stream by applying visual effects to the first video stream and sends the second video stream to the server (or another server). For example, in the first mode, the client device sends a raw video stream without visual effects to the server, and further sends visual effects information regarding the visual effects to be applied to the first video stream. For example, the visual effect could be a blur effect to be applied to the background of a self-view video stream. The server receives the raw video stream and, based on the visual effects information, applies the blur effect to the background of the self-view video stream to generate a modified video stream. The server sends the corrected video stream to other client devices participating in the video conference session, and, if necessary, also sends the corrected video stream to the client device that originally sent the raw video stream to the server.

[0034] According to embodiments disclosed herein, visual effects may be applied to a video stream by one or more servers referred to as video processing servers, as described above. For example, a video processing server may receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to uncompressed media. To implement visual effects on a video stream, a video processing server may utilize libraries that support combining visual rendering effects (e.g., blurring, scaling, brightness, etc.) with foreground segmentation (e.g., segmenting the user in the user's self-view on a client device and applying visual effects to the background). A video processing server may be specifically configured to perform video processing on a video stream received from a client device in order to apply visual effects to the video stream. For example, a video processing server may include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or for applying visual effects to a video stream), and multiple video encoder accelerators (e.g., application-specific integrated circuits (ASICs) for performing transcoding operations such as encoding, decoding, and resizing).

[0035] According to embodiments disclosed herein, a loopback stream is implemented from the video processing server to the client device so that a user of the client device can see the visual effects applied to the video stream. In exemplary embodiments, one or more intermediate servers may be placed between the client device and the video processing server, so that one or more intermediate servers route the video stream from the client device to the video processing server, and one or more intermediate servers route the modified video stream from the video processing server to the client device.

[0036] The video processing server is configured to perform video transcoding operations. For example, the video processing server may receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to the uncompressed media. To apply visual effects to a video stream, the video processing server may utilize libraries that support visual rendering effects (e.g., blur, scaling, brightness, etc.) in combination with foreground segmentation (for example, segmenting the user in the user's self-view on the client device and applying visual effects to the background).

[0037] For example, when a client device applies visual effects to a video stream itself, the video processing server may be bypassed, and the video stream may be sent to the client device participating in the video conferencing session via another server in the computing network (server system). In another exemplary embodiment, when it is not required or necessary for the video processing server to apply visual effects to the input video stream, the video processing server is configured to decode the video stream received from the client device and pass the decoded video stream to one resizer node, i.e., multiple resizer nodes, for each specific resolution requested by each client device receiving the video stream of the video conferencing session. That is, the video processing server is configured to perform video transcoding operations using a graph of nodes. In some embodiments where the video processing server does not apply visual effects, the video stream does not pass through the video processing server because it is not necessary for the video processing server to perform resizing operations, and a client device requesting the highest resolution video stream may receive the video stream from the client device via another server. In another embodiment, the video processing server is configured to pass the decoded video stream to multiple resizer nodes, each of which is bucketed to correspond to a specific bitrate resolution for each client device participating in the video conferencing session, although it may or may not correspond precisely to the resolution of each client device. For example, each client device may have the capability to process a high-definition (HD) video stream (e.g., 1920x1080), but may be limited to a specific resolution (e.g., 800x600) due to bitrate and / or bandwidth constraints, so the video processing server may resize the modified video stream according to a specific resolution based on the bitrate of each client device.

[0038] According to exemplary embodiments disclosed herein, a client device is configured to request a video processing server to apply one or more visual effects to a video stream that the client device generates and transmits to the video processing server (e.g., via an intermediate server). The client device may be configured to signal the video processing server to apply visual effects to the video stream, for example, through configuration properties associated with the video stream (e.g., via metadata). The configuration properties may include one or more settings for one or more visual effects to be applied to the video stream by the video processing server. When the client device requests the video processing server to apply visual effects to the video stream, the video processing server is configured to decode the video stream received from the client device, pass the decoded video stream to a visual effects applicator (also referred to as a media pipe effects module), apply the visual effects to the video stream, and generate a modified video stream. The modified video stream is then passed to one resizer node for each specific resolution requested by each client device that receives the modified video stream, and thus to multiple resizer nodes. In another embodiment, the video processing server is configured to pass the corrected video stream to multiple resizer nodes, each of which is bucketed to correspond to a specific bitrate resolution for each client device participating in the video conferencing session, although it may or may not correspond precisely to the resolution of each client device.

[0039] According to exemplary embodiments disclosed herein, a client device may be configured to receive a modified video stream from a video processing server. The client device may specifically request the video processing server to return the modified video stream to the client device, or the video processing server may be configured to automatically return the modified video stream to the client device. For example, when a visual effect is requested by the client device, the video processing server is configured to signal (e.g., via a message or via metadata) to an intermediate server (i.e., a server located between the client device and the video processing server that forwards the video stream received from the client device to the video processing server) to use at least a portion of the video stream received from the client device and send it from the video processing server to the client device via the intermediate server. Thus, in response to receiving the signal from the video processing server, the intermediate server prepares to return at least a portion of the video stream to the client device, and a push notification is sent to the client device informing it of the new stream. When the client device receives the notification about the modified stream, the client device is configured to perform a lookup of the input data from the intermediate server, including the modified video stream. Client devices may be configured to prioritize displaying the corrected video stream over local streams generated on the client device.

[0040] According to exemplary embodiments disclosed herein, a client device may be configured to toggle or switch between a first mode in which the client device requests a video processing server to apply visual effects to a video stream generated by the client device, and a second mode in which the client device applies the visual effects to the video stream and sends the modified video stream to the video processing server. For example, if a user of the client device switches to the first mode in which the video processing server applies background blur to the video stream while a background replacement visual effect is applied to the video stream, one or more processors of the client device may be configured to first enable the first mode, thereby allowing the video processing server to apply background blur to the video stream and, at the same time, still temporarily apply the background replacement visual effect until the background replacement visual effect is disabled. Thus, several frames of the video stream may contain both visual effects, but such a method ensures that unprocessed frames are not transmitted to other client devices participating in the video conferencing session.

[0041] According to exemplary embodiments disclosed herein, a client device may be configured to distinguish between a “waiting room” state, where the user is waiting to be allowed to join or to join a video conference meeting, and a “joined” state, where the user is allowed to join or has joined the meeting, when switching between a first mode and a second mode. For example, in response to the client device transitioning from the waiting room to the joined state, one or more processors in the client device may be configured to switch from the second mode to the first mode, thereby making it the responsibility of the video processing server to modify the video stream generated by the client device with visual effects. For example, background blur effects (or other visual effects) applied to the video stream by the client device while the client device is waiting in the waiting room to be allowed to join the meeting are “transferred” from the client to the video processing server when the client device is allowed to join the meeting. Thus, when the client device joins the meeting, it is the responsibility of applying background blur effects (or other visual effects).

[0042] According to exemplary embodiments disclosed herein, a video conferencing application may be set to a first mode by default. That is, the video conferencing application may, by default, prefer that the video processing server apply visual effects in the first mode (rather than the client device in a second mode) during a video conferencing session. In response to the video processing server being unavailable (e.g., due to insufficient resources, high peak usage, bandwidth limitations, etc.), one or more processors in the client device may be configured to switch to a second mode, thereby allowing the client device to apply the processing of visual effects. For example, in response to insufficient graphics computing resources on the video processing server, the intermediate server may be configured to signal back to the client device (e.g., via a message or via metadata) that the first mode (i.e., visual effects are applied by the video processing server rather than the client device) is unavailable, and the client device falls back to a second mode in which the client device applies visual effects (i.e., client-side effects). In an exemplary embodiment, if the client device does not support a visual effect requested to be applied to a video stream, one or more processors in the client device may be configured to control the client device's display to show a message indicating that the requested visual effect cannot be applied to the video stream, and the client device's camera may be muted to protect user privacy. The client device may also notify the user that the requested visual effect cannot be applied to the video stream in an additional or alternative way (e.g., via the speaker).

[0043] In an exemplary embodiment, a client device may request a video processing server to apply visual effects to a video stream, and if the video processing server is unable to modify the video stream (e.g., due to insufficient resources or processing limitations), the video processing server may be configured to forward the video stream to other client devices in the video conference session without applying the visual effects. However, in an exemplary embodiment, if the video processing server is unable to modify the video stream (e.g., due to insufficient resources or processing limitations), the video processing server may also be configured not to forward the video stream to other client devices in the video conference session. If the video processing server is unable to modify the video stream as requested, the video processing server may receive a command from the client device not to forward the video stream to other client devices in the video conference session. If the requested visual effects are related to privacy (e.g., if a user does not want other users in the video conference session to see their unblurred background), this embodiment provides further privacy to the user of the client device. Thus, a remote client device in the video conference session may receive the modified video stream, or, during intermittent failures, may not receive any video at all.

[0044] According to exemplary embodiments disclosed herein, a client device may be configured to display a self-view of the user of the client device during a video conferencing session. For example, when the client device operates in a first mode and requests a video processing server to apply visual effects to a video stream, the client device may be configured to receive the modified video stream "remotely." The modified video stream received remotely may have worse quality and latency compared to a local self-view generated by the client device due to network round-tripping and transcoding performed by the video processing server. However, according to exemplary embodiments disclosed herein, the quality and latency of the self-view can be improved according to various techniques. Thus, the media quality and latency of the self-view can be maintained at a level that does not adversely affect the user's perception of the quality of the video conferencing session. For example, to achieve lower latency (e.g., a threshold of less than 1 second, a threshold of less than 300 mms, etc.), the video processing server may be configured to enable WebRTC low-latency rendering and set latency threshold limits when sending the corrected video stream back to the client device (e.g., by setting threshold limits in the PlayoutDelayLimits RTP header extension). As an additional or alternative technique to improve self-view quality, the video processing server may be configured to adjust its bitrate allocation strategy to prioritize sending the corrected video stream back to the client device rather than sending it to other client devices. That is, the video processing server may be configured to prioritize corrected video streams to be looped back when distributing available bandwidth downstream. If this prioritization is not performed, in larger meetings, a user on a client device may lose their self-view when not speaking, while other users in the video conference session can still see the user.As an additional or alternative technique to improve self-view quality, one or more processors in the client device may be configured, as a performance adaptive, to reduce the frame rate of remotely received video streams (e.g., video streams sent from other client devices during a video conference session), but the corrected video stream for self-view is excluded (i.e., maintained at the default frame rate). If, despite one or more of the aforementioned techniques being implemented (e.g., by prioritizing the corrected video stream sent back to the client device), there is still insufficient bandwidth for the corrected video stream to be looped back, one or more processors in the client device may be configured to detect that the video processing server and / or intermediate server cannot correct the video stream generated by the client device with visual effects, and the client device may switch from processing the video in a first mode to processing the video in a second mode, and the client device corrects the video stream by applying visual effects to the video stream.

[0045] According to exemplary embodiments disclosed herein, a client device may be configured to toggle or switch between a first mode in which the client device requests a video processing server to apply visual effects to a video stream generated by the client device, and a second mode in which the client device applies visual effects to the video stream and sends the corrected video stream to the video processing server, based on the size of the self-view and / or the physical resolution of the client device's display. The size of the self-view may correspond to the active resolution of the self-view. For example, during a video conferencing session, if the size of the self-view displayed on the client device's display exceeds a threshold, one or more processors in the client device may be configured to switch to (or maintain) the second mode in which the client device applies visual effects to the video stream and sends the corrected video stream to the video processing server. When the size of the self-view exceeds a threshold, a video quality problem perceptible to the user may occur on the client device. Conversely, when the size of the self-view displayed on the client device's display is below a threshold, one or more processors in the client device are configured to switch to (or maintain) a first mode in which the client device requests that the video processing server apply visual effects to the video stream generated by the client device.

[0046] In one embodiment, when multiple self-views are displayed on the client device's display during a video conferencing session, one or more processors in the client device are configured to identify the self-view with the largest size. One or more processors in the client device are configured to determine whether the size of the self-view with the largest size exceeds a threshold, and whether the client device switches to (or maintains) a second mode in which it applies visual effects to the video stream and sends the modified video stream to the video processing server. The size of the self-view may correspond to the active resolution of the self-view with the largest size. One or more processors may be configured to track or monitor the size of one or more self-views during the video conferencing session to determine whether there is a change in the maximum size and whether the client device switches to (or maintains) a second mode in which it applies visual effects to the video stream and sends the modified video stream to the video processing server. Changes to the size and / or position of the self-views may be sent to the video processing server to update the position of the visual effects applied to the self-views, depending on whether the client device has switched to the first mode or the second mode. In the embodiment, when the size of the self-view is frequently changed by the user, one or more processors may be configured to wait until the resizing stops for a predetermined time (e.g., 3 seconds) before determining whether the size of the self-view displayed on the client device's display exceeds a threshold.

[0047] Intermittent video problems (such as video freezing or frame drops) may occur when a client device toggles or switches between a first mode in which the client device requests the video processing server to apply visual effects to a video stream generated by the client device, and a second mode in which the client device applies visual effects to the video stream and sends the modified video stream to the video processing server. Such intermittent video problems may be even more apparent in a self-view than in a view by a remote participant in a video conferencing session. In an exemplary embodiment, when switching from one embodiment to another (i.e., from the first mode to the second mode, or vice versa), the client device is configured to wait until the new embodiment has started before shutting down the previous embodiment (i.e., until the client device begins receiving remote frames of the self-view generated by the video processing server, or until the client device applies visual effects).

[0048] For example, when switching from a second mode to a first mode, one or more processors in the client device may be configured to continue displaying a self-view via the local stream until the first keyframe of the remotely corrected video stream is received from the video processing server. This technique eliminates freezes and minimizes the number of frames with double effects applied (e.g., double-blurred frames with visual effects applied by both the client device and the video processing server). If the video stream has not yet been resized to accommodate multiple resolutions, the remote view on other client devices during the video conferencing session may freeze when a visual effect is requested from the video processing server, and recover from the freeze once the first keyframe is propagated from the client device through the video processing server. As the client device continues to apply visual effects until it encounters a new corrected video stream, some frames with double effects will result. Both client devices participating in the video conferencing session and remote client devices may experience quality degradation due to the resumption of the video stream.

[0049] For example, when switching from the first mode to the second mode, one or more processors in the client device may be configured to immediately display a self-view using the local stream once the client applies visual effects, without any artifacts being perceived. Remote views on other client devices during a video conferencing session may display some frames with double effects applied (e.g., double blur) until the video processing server disables the visual effects, and may also display some stalls / frame drops while the client device is loading the visual effects.

[0050] In some embodiments, other configurations may be implemented in the client device and / or the video processing server to reduce freeze time and / or reduce or avoid double-effect frames. For example, the video processing server may be kept in a state ready to apply visual effects to the video stream sent from the client device (e.g., a ready state), but does not actually apply any visual effects until the mode is switched to a first mode. By keeping the video processing server ready to apply visual effects, the startup time of the video processing server is reduced, and the freeze time of the remote client device during a video conferencing session is reduced. In embodiments, the client device may be configured to send a timestamp / frame counter to the video processing server indicating a future time when the client device wants to switch modes. The video processing server is configured to ensure that the visual effects are enabled / disabled in that frame at the appropriate time. The client device is configured to wait until the value of the timestamp / frame counter is reached before starting / stopping the application of visual effects, thus preventing double-effect frames. According to this embodiment in which the client device sends a timestamp / frame counter, the video processing server may be configured to switch modes immediately and / or be kept in a ready state.

[0051] In this embodiment, the client device may be configured to transmit metadata for frames of the video stream, which provides the video processing server with information about visual effects. When the client device requests the video processing server to apply visual effects, the video processing server is configured to generate a modified video stream with visual effects only if the visual effects to be applied are different from any visual effects applied to the input video stream. According to this embodiment, where the client device transmits metadata for frames of the video stream that provides the video processing server with information about visual effects, the video processing server may be configured to switch modes immediately and / or remain in a ready state.

[0052] For example, in some embodiments, the client device and / or the video processing server may preload visual effects without actually applying them to the output stream. For instance, when switching from a first mode to a second mode, the client device may be configured to preload the visual effects, request the video processing server to freeze the current frame to which the visual effects have been applied by the video processing server, send the video stream to which the visual effects have been applied by the client device, and then request the video processing server to stop applying the visual effects to the video stream. Thus, instead of a double-effect frame, a short video freeze is added. For example, when switching from a second mode to a first mode, the video processing server may be configured to receive a request from the client device to preload the visual effects and freeze the frame to which the visual effects have been applied by the client device. The client device may stop applying the visual effects to the video stream and send the video stream without the visual effects to the video processing server. The video processing server is then requested to apply the visual effects to the video stream. Thus, instead of a double-effect frame, a short video freeze is added.

[0053] In exemplary embodiments, a mixed operation mode may be implemented between a client device and a video processing server. For example, when a client device switches to applying visual effects to a video stream, one or more processors in the client device may be configured to send the unmodified video stream to the video processing server while simultaneously displaying a self-view with the video stream to which the client device has applied visual effects. This allows the video processing server to continue applying visual effects to the unmodified video stream, and the modified video stream (the video stream to which the video processing server has applied visual effects) is sent for display on remote client devices participating in the video conferencing session. In this technique, the remote client device is not aware of the switch from the first mode, however, a temporary glitch may occur in the self-view on the client device due to the different video stream timelines when switching to mixed mode. In some implementations, to avoid running in mixed mode for extended periods, the client device may be configured to switch from mixed mode to a second mode after a predetermined period of time. When the self-view is determined to be below a threshold size, this technique prevents the waste of cloud resources, allowing client devices to apply visual effects without excessively degrading the self-view.

[0054] In another exemplary embodiment, to speed up the switching from the second mode to the first mode in SelfView, the view requests in SelfView may be limited to a threshold resolution (e.g., 640 x 360).

[0055] In another exemplary embodiment, switching to the first mode may be disabled when any zoom function (e.g., auto-zoom or centering function) is enabled in the video conferencing application on the client device. However, in the embodiment, when the zoom function is enabled in the video conferencing application on the client device, the video processing server may be configured to apply visual effects to the video stream in the first mode. For example, an enlarged stream (i.e., a raw video stream modified with zoom on the client) may be sent from the client device to the video processing server, and visual effects may be applied to the enlarged stream. For example, a raw video stream may be sent from the client device to the video processing server, and the video processing server may apply both enlargement and visual effects to the video stream.

[0056] This disclosure offers numerous technical effects and benefits. Video processing is a costly processing task (e.g., a CPU / GPU-intensive task) that can consume a significant amount of bandwidth on client devices such as laptops, tablets, and smartphones. By running video conferencing applications and simultaneously adding visual effects to a video stream, client devices may overheat, experience battery drain, and / or struggle to implement high-quality visual effects (e.g., reduced frames per second or slower client devices). As an example of the technical effects and benefits, the systems and methods disclosed herein enable offloading certain video processing to the cloud (e.g., to a server such as a video processing server), thereby relieving pressure on the client device. Thus, by providing a server such as a video processing server, which consists of hardware that modifies the video stream sent by the client device to include the visual effects requested by the client device, it becomes possible to smoothly run high-quality and efficient video conferencing on the client device. Technical benefits include reduced client CPU usage, improved stability, increased bandwidth, improved video quality (e.g., higher resolution and / or frame rate), and an improved user experience.

[0057] Referring here to the drawings, Figure 1 is an exemplary video conferencing system according to one or more exemplary embodiments of the present disclosure. Figure 1 shows an embodiment of a video conferencing system including a first client device 100, a second client device 100', a third client device 100'', a first server (first intermediate server) 200, a second server (second intermediate server) 200', a third server (third intermediate server) 200'', and a video processing server 300, each of which can communicate with each other over a network. For example, the first client device 100, the second client device 100', and the third client device 100'' may include any of the following: a personal computer, a smartphone, a laptop, and a tablet computer. The network may include any type of communication network, such as a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), or a virtual private network (VPN). For example, wireless communication between elements of an exemplary embodiment may be performed via Wi-Fi, Wi-Fi, Bluetooth®, ZigBee®, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), and radio frequency (RF) signals. For example, wired communication between elements of an exemplary embodiment may be performed via paired cables, coaxial cables, fiber optic cables, and Ethernet® cables.

[0058] As will be explained in more detail below, in some embodiments, the first client device 100 and one or more other client devices (e.g., a second client device 100', a third client device 100', etc.) may participate in a video phone (hereinafter referred to as a video conference) session using a video processing server 300 over a network. A video conference session may include multiple client devices, including client devices other than the first client device 100, the second client device 100', and the third client device 100'' shown in Figure 1.

[0059] In Figure 1, the first client device 100 transmits a video stream 190 (e.g., a raw video stream) to the video processing server 300 via the first server 200. The video stream 190 may correspond to a self-view of the user of the first client device 100 who is participating in or waiting to participate in a video conference session with other client devices. In this embodiment, the video stream 190 may be a raw video stream to which no visual effects have been applied (e.g., no background blur and / or brightness effects have been applied to the user's self-view). In addition to the video stream 190, in this embodiment, the first client device 100 transmits visual effect information 190' (e.g., media features) to the video processing server 300 via the first server 200, which corresponds to the visual effects applied to the video stream by the video processing server 300.

[0060] In response to receiving the video stream 190 and the visual effects information 190', the video processing server 300 modifies the video stream 190 by adding one or more visual effects to the video stream 190 based on the visual effects information 190'. The video processing server 300 sends the modified video stream to other client devices participating in the video conferencing session. For example, the video processing server 300 sends the modified video stream 192 to a second client device 100' via a second server 200', and the modified video stream 194 to a third client device 100' via a third server 200''. In some embodiments, the video processing server 300 may send the modified video stream (loopback stream) 196 back to the first client device 100 via the first server 200. In some embodiments, the modified video streams 192, 194, and 196 may be the same (e.g., having the same resolution), while in other embodiments, the modified video streams 192, 194, and 196 may be different from each other (e.g., having different resolutions). Further details regarding the operation of the video processing server 300 are discussed below.

[0061] Figures 2A and 2B show exemplary flowcharts for a video processing server according to one or more exemplary embodiments of the present disclosure. For example, when a first client device 100 applies visual effects to a video stream itself, the video processing server 300 may be bypassed, and the video stream may be transmitted to client devices participating in the video conferencing session using other servers in the computing network (server system). In another exemplary embodiment, when it is not required or necessary for the video processing server 300 to apply visual effects to an input video stream, the video processing server 300 is configured to decode the video stream received from the first client device 100 and to pass the decoded video stream to one resizer node (resizer), i.e., multiple resizer nodes (resizers), for each specific resolution requested by each client device receiving the video stream of the video conferencing session. That is, the video processing server 300 is configured to perform video transcoding operations using a graph of nodes. Transcoding refers to operations such as decompressing a video stream (e.g., decoding), modifying the decompressed video stream (e.g., applying one or more visual effects, resizing the video stream), and recompressing the video stream (e.g., encoding). In embodiments where the video processing server 300 does not apply visual effects, the video stream does not pass through the video processing server 300 (because the video processing server 300 does not need to perform a resizing operation), and a client device requesting the highest resolution video stream may receive the video stream from the client device via another server. According to another embodiment, the video processing server is configured to pass the decoded video stream to a number of resizer nodes, each of which is bucketed to correspond to a specific bitrate resolution for each client device participating in the video conferencing session, although it may or may not correspond precisely to the resolution of each client device.For example, each client device may have the capability to process a high-resolution (HD) video stream (e.g., 1920 x 1080), but may be limited to a specific resolution (e.g., 800 x 600, 640 x 360, etc.) due to bitrate and / or bandwidth constraints. Therefore, the video processing server 300 may resize the video stream according to a specific resolution based on the bitrate of each client device.

[0062] Referring to Figure 2A, the video processing server 300 may be configured to process an input video stream 210 received from a first client device 100, the input video stream 210 having visual effects applied on the client side (i.e., by the first client device 100). In the embodiment of Figure 2A, the video processing server 300 processes the input video stream 210 with a decoder 332 that decodes the input video stream 210. The video processing server 300 is also configured to resize the decoded video stream with resizers 334a, 334b ​​that perform resizing operations to obtain a desired resolution for the input video stream 210. As described above, the desired resolution may be obtained based on the capabilities of the client device receiving the video stream from the video processing server 300, or based on existing bitrate and / or bandwidth constraints relating to the client device receiving the video stream from the video processing server 300. For example, resizer 334a may be configured to resize the decoded video stream to a first resolution (e.g., 640 x 480), and resizer 334b ​​may be configured to resize the decoded video stream to a second resolution (e.g., 320 x 240). Encoder 336a is configured to encode the video stream resized by resizer 334a and output a first output video stream 220a, and encoder 336b is configured to encode the video stream resized by resizer 334b ​​and output a second output video stream 220b. For example, the first output video stream 220a may be sent to a second client device 100' via a second server 200', and the second output video stream 220b may be sent to a third client device 100' via a third server 200''.

[0063] Referring to Figure 2B, the video processing server 300 is shown to include a visual effects applicator 338. The visual effects applicator 338 is also included in the embodiment of Figure 2A, but is omitted for clarity. In the embodiment of Figure 2A, the visual effects have already been applied by the first client device 100, and the video stream is simply passed to the resizers 334a, 334b, so the visual effects applicator 338 may be bypassed or unable to perform any action with respect to the video stream. In the embodiment of Figure 2B, the video processing server 300 may be configured to process an input video stream 230 received from the first client device 100, the input video stream 230 has not had any visual effects applied on the client side (i.e., by the first client device 100), and instead, the visual effects applicator 338 applies one or more visual effects to the input video stream 230 (i.e., on the cloud side). In the embodiment shown in Figure 2B, the video processing server 300 processes the input video stream 230 using a decoder 332 that decodes the input video stream 230. The visual effects applicator 338 is configured to apply one or more visual effects to the decoded video stream according to visual effects information 230' also received from the first client device 100. The visual effects information 230' may include information about one or more settings for the visual effects to be applied to the input video stream 230. For example, one or more settings may include a blur radius for a background blur effect to be applied to at least a portion of the input video stream 230. For example, one or more settings may include a brightness level for a contrast effect to be applied to at least a portion of the input video stream 230. Similar to Figure 2A, the video processing server 300 is configured to resize the decoded video stream (with visual effects applied by the visual effects applicator 338) using one or more resizers. In the embodiment shown in Figure 2B, three resizers 334a, 334b, and 334c are shown. The resizers 334a, 334b, and 334c are configured to perform a resizing operation to obtain the desired resolution of the input video stream 230.The desired resolution may be obtained based on the ability of the client device to receive the video stream from the video processing server 300, or based on existing bitrate and / or bandwidth constraints relating to the client device receiving the video stream from the video processing server 300. For example, resizer 334a may be configured to resize the decoded video stream to a first resolution (e.g., 640×480), resizer 334b ​​may be configured to resize the decoded video stream to a second resolution (e.g., 320×240), and resizer 334c may be configured to resize the decoded video stream to HD resolution (e.g., 1920×1080). Encoder 336a is configured to encode the video stream resized by resizer 334a and output a first output video stream 240a; encoder 336b is configured to encode the video stream resized by resizer 334b ​​and output a second output video stream 240b; and encoder 336c is configured to encode the video stream resized by resizer 334c and output a third output video stream 240c. For example, the first output video stream 240a may be sent to a second client device 100' via a second server 200', the second output video stream 240b may be sent to a third client device 100' via a third server 200'', and the third output video stream 240c may be sent to the first client device 100 via the first server 200. However, this disclosure is not limited to these exemplary resolution values, and in some embodiments, the resolution of the video stream received from the first client device 100 may not be changed or resized. Furthermore, this disclosure is not limited to the exemplary client devices described as receiving the output video stream, and the output video stream may be sent to client devices participating in the video conferencing session other than the exemplary client devices described.

[0064] Referring here to Figure 3, an exemplary block diagram of a client device and a video processing server according to one or more exemplary embodiments of the present disclosure is described herein. Figure 3 shows a first client device 100, but the features of the first client device 100 described herein are also applicable to a second client device 100' and a third client device 100''.

[0065] The first client device 100 may include one or more processors 110, one or more memory devices 120, a video conferencing application 130, a camera 140, an input device 150, and a display 160. The video processing server 300 may include one or more processors 310, one or more memory devices 320, and a video conferencing service provider 330.

[0066] For example, one or more processors 110, 310 may be any suitable processing device that may be included in the first client device 100 or the video processing server 300. For example, such processors 110, 310 may include one or more of the following: processors, processor cores, controllers and arithmetic logic units, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), image processors, microcomputers, field-programmable arrays, programmable logic units, application-specific integrated circuits (ASICs), microprocessors, microcontrollers, and any other devices that can be executed in response to instructions in a defined manner, and in combination thereof. One or more processors 110, 310 may be a single processor or multiple processors that are operablely connected, for example, in parallel.

[0067] The memories 120 and 320 may include one or more non-temporary computer storage media, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and volatile memory devices such as flash memory, USB drives, and random access memory (RAM), as well as optical media such as hard disks, floppy disks (registered trademarks), Blu-ray discs, or CD-ROM discs and DVDs, and combinations thereof. However, embodiments of the memories 120 and 320 are not limited to the above description, and as will be understood by those skilled in the art, the memories 120 and 320 may be implemented by a variety of other devices and structures.

[0068] For example, memory 120 may store instructions, and when these instructions are executed, they cause one or more processors 110 to execute a video conferencing application 130 with another client device via a video processing server 300, as described in embodiments of the present disclosure. For example, memory 320 may store instructions, and when these instructions are executed, they cause one or more processors 310 to provide a video conferencing service to multiple client devices, as described in embodiments of the present disclosure.

[0069] Memory 120 may also include data 122 and instructions 124, which may be retrieved, manipulated, created, or stored by one or more processors 110. In some exemplary embodiments, as described in embodiments of this disclosure, such data may be accessed and used as input to run a video conferencing application 130 with another client device via a video processing server 300. Memory 320 may also include data 322 and instructions 324, which may be retrieved, manipulated, created, or stored by one or more processors 310. In some exemplary embodiments, as described in embodiments of this disclosure, such data may be accessed and used as input to provide video conferencing services to multiple client devices.

[0070] The first client device 100 includes a video conferencing application 130, which may also be referred to as a video phone application or video call application. The video conferencing application 130 enables a user of the first client device 100 to communicate with a user of another client device via the transmission of audio and visual signals, thereby enabling two or more users to participate in a real-time conversation from different locations and enable two or more users to view each other via a video stream from different locations. According to embodiments of this disclosure, the video conferencing application 130 may include a self-view size specifier 132 and a visual effects mode selector 134. The features of the self-view size specifier 132 and the visual effects mode selector 134 are described in more detail below.

[0071] The first client device 100 includes a camera 140, and the video conferencing application 130 can capture image data from the camera 140 and generate a video stream to be sent to the video processing server 300. In some exemplary embodiments, the camera 140 collects image data, for example, self-views of one or more users of the first client device 100. The camera 140 can be any device capable of capturing visual data. The first client device 100 is configured to generate a video stream and send the video stream to the video processing server 300 (for example, via the network 400 and one or more intermediate servers, such as the first server 200). For example, the camera 140 may be an integrated webcam of the first client device 100, or it may be a camera device connected in a communicative manner, etc. The first client device 100 can encode the captured video (for example, as specified in instruction 124, etc.). In some embodiments, the first client device 100 can encode the captured video in high resolution and high quality (for example, in HD format).

[0072] The first client device 100 includes an input device 150 configured to receive input from a user, which may include one or more of the following: a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, buttons, switches, an electronic pen or stylus, a gesture recognition sensor (e.g., recognizing user gestures including the movement of body parts), an input sound device or voice recognition sensor (e.g., a microphone that receives voice commands), a trackball, a remote controller, a mobile phone (e.g., a cellular phone or smartphone), a tablet PC, a pedal or foot switch, and a virtual reality device. The input device 150 may further include a haptic device that provides haptic feedback to the user. The input device 150 may also be embodied, for example, by a touch-sensitive display having touchscreen functionality. The input device 150 may be used by a user of the first client device 100 to select one or more visual effects to apply to a video stream generated by the first client device 100. For example, the input device 150 may be used by the user of the first client device 100 to select one or more visual effects to apply to the user's self-view, such as contrast effects, background blur effects, and background replacement effects. The input device 150 may also be used by the user of the first client device 100 to select default settings regarding when visual effects are applied on the client side and when they are applied on the cloud side. For example, the user can specify a default setting such that the video processing server 300 applies visual effects to the video conferencing session when the video conferencing application 130 is first run.

[0073] The first client device 100 includes a display 160 that displays information visible to the user. For example, the display 160 may be a non-touch-sensitive display or a touch-sensitive display. The display 160 may include, for example, liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, active-matrix organic light-emitting diode (AMOLED) displays, flexible displays, 3D displays, plasma display panels (PDPs), and cathode ray tube (CRT) displays. However, this disclosure is not limited to these exemplary displays and may include other types of displays.

[0074] According to the exemplary embodiments described herein, the video processing server 300 may include one or more processors 310 and memory 320 as described above. The video processing server 300 may also include a video conferencing service provider 330. For example, the video conferencing service provider 330 may include a transcoder 331 and a visual effects applicator 338. The transcoder 331 may be configured to receive compressed media (e.g., video) from a first client device 100, decompress the media, and apply visual effects to the uncompressed media. To implement visual effects on the video stream, the transcoder 331 may utilize a library that supports visual rendering effects (e.g., blur, scaling, brightness, etc.) in combination with foreground segmentation (e.g., segmenting the user in the user's self-view on the first client device 100 and applying visual effects to the background). For example, the transcoder 331 may include one or more decoders 332 that decode a video stream received from a first client device 100, as discussed with respect to embodiments, for example, Figures 2A to 2B. For example, the transcoder 331 may include one or more resizers 334 that resize a video stream received from a first client device 100, as discussed with respect to embodiments, for example, Figures 2A to 2B. For example, the transcoder 331 may include one or more encoders 336 that encode each video stream resized by the corresponding resizer 334, as discussed with respect to embodiments, for example, Figures 2A to 2B, and the encoded video streams are then output to each client device participating in the video conferencing session.

[0075] The visual effects applicator 338 is configured to apply one or more visual effects to the video stream received from the first client device 100 based on visual effects information received from the first client device 100. For example, the visual effects information may be received in the form of configuration properties associated with the video stream (e.g., via metadata). The configuration properties may include one or more settings for one or more visual effects to be applied to the video stream by the video processing server 300. When the first client device 100 requests the video processing server 300 to apply visual effects to the video stream, the decoder 332 decodes the video stream received from the first client device 100, passes the decoded video stream to the visual effects applicator 338 (also referred to as the media pipe effects module), and the visual effects applicator 338 applies the visual effects to the video stream to produce a modified video stream.

[0076] The video processing server 300 may be a specially configured server or group of servers dedicated to performing video processing on video streams received from client devices participating in a video conferencing session in order to apply visual effects to the video stream. For example, the video processing server 300 may include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or for applying visual effects to the video stream), and multiple video encoder accelerators (e.g., ASICs for performing transcoding operations such as encoding, decoding, and resizing).

[0077] Further embodiments of the first client device 100 and the video processing server 300 will be discussed with reference to the following examples shown in Figures 4 to 5 and the flowcharts in Figures 6 to 10.

[0078] Figure 4 shows an exemplary display of a video conferencing session on a client device according to one or more exemplary embodiments of the present disclosure. Referring to Figure 4, during a video conferencing session 410, while the first client device 100 is running the video conferencing application 130, the display 160 may display a presentation 420, a self-view 430 of the user of the first client device 100, and one or more external live streams 440 of one or more other users of other client devices participating in the video conferencing session 410. For example, the first client device 100 may receive one or more video streams from the video processing server 300, the first server 200, or another server not specifically shown in the drawings. The position and / or size of the presentation 420, the self-view 430, and / or external live streams 440 may be changed, for example, according to input received from the user of the first client device 100. For example, the user of the first client device 100 may resize the self-view 430 via the input device 150. Furthermore, one or more of the presentation 420, the self-view 430, and one or more external live streams 440 do not have to be displayed on the display 160 of the first client device 100. Also, although Figure 4 shows that the user of the first client device 100 is the presenter of the presentation 420, the user of the first client device 100 may be any participant in the video conferencing session 410 and may not be the presenter.

[0079] Figure 5 shows an exemplary display in which a visual effect is applied to a user's self-view during a video conferencing session on a client device, according to one or more exemplary embodiments of the present disclosure. Referring to Figure 5, during a video conferencing session, while the video conferencing application 130 is running on the first client device 100, the display 160 may display the user's self-view 510 on the first client device 100, the self-view 510 including the user 520 and a blurred background 530. Thus, in the exemplary self-view 510 of Figure 5, a background-blurring visual effect is applied to the background of the user 520, for example, the user 520 may have selected or requested this visual effect for privacy reasons. Depending on the operating mode of the first client device 100, depending on the size of the user's self-view, and according to various other factors described later, the client device 100 or the video processing server 300 may apply a visual effect to the background. The present disclosure is not limited to the exemplary embodiments of Figure 5. For example, visual effects other than the blurring effect, or additional visual effects in addition to the blurring effect, may be applied to the video stream. For example, light and dark effects (such as increasing or decreasing brightness and / or contrast values), replacement backgrounds, and other visual effects may be applied to the video stream.

[0080] Figures 6 to 10 show flowcharts of exemplary and non-limiting computer implementations according to one or more exemplary embodiments of the present disclosure.

[0081] Referring to Figure 6, Method 600 includes Operation 610, in which a server system (e.g., video processing server 300) receives a video stream relating to a video conferencing session from a first client device 100. The Method further includes Operation 620, in which the server system (e.g., video processing server 300) receives visual effects information relating to one or more visual effects to apply to the video stream from the first client device 100. Operation 630 includes the server system (e.g., video processing server 300) applying one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams (e.g., modified video streams 192, 194). Operation 640 includes the server system (e.g., video processing server 300) sending one or more modified video streams to one or more other client devices (e.g., a second client device 100' and / or a third client device 100'') participating in the video conferencing session. Although not shown in Figure 6, optional operations may include the server system (e.g., video processing server 300) applying one or more visual effects to the video stream based on the received visual effects information to generate a modified video stream (e.g., a loopback or modified video stream 196), and then sending the modified video stream to the first client device 100.

[0082] With respect to the modified video stream 196 (i.e., the loopback video stream), the first client device 100 may be configured to receive the modified video stream 196 from the video processing server 300 by specifically requesting the video processing server 300 to send the modified video stream 196 back to the first client device 100, or the video processing server 300 may be configured to automatically send the modified video stream 196 to the first client device 100. For example, when a visual effect is requested by the first client device 100, the video processing server 300 is configured to signal (e.g., via a message or via metadata) to an intermediate server (e.g., the first server 200) to also send from the video processing server 300 to the client device 100 via the intermediate server using at least a portion of the video stream received from the first client device 100. Therefore, in response to receiving a signal from the video processing server 300, the intermediate server prepares to return at least a portion of the video stream to the first client device 100, and a push notification is sent to the first client device 100 informing it of the new stream. When the first client device 100 receives the notification about the corrected stream, the first client device 100 is configured to retrieve input data from the intermediate server, including the corrected video stream 196. The first client device 100 may be configured to preferentially display the corrected video stream 196 over the video stream 190 generated locally on the first client device 100.

[0083] In an exemplary embodiment, the first client device 100 requests the video processing server 300 to apply one or more visual effects to the video stream 190, and if the video processing server 300 is unable to modify the video stream 190 (e.g., due to insufficient resources or processing limitations), the video processing server 300 may be configured to forward the video stream 190 without applying any visual effects to other client devices in the video conferencing session (e.g., a second client device 100' and / or a third client device 100''). However, in an exemplary embodiment, the video processing server 300 may also be configured not to forward the video stream 190 to other client devices in the video conferencing session (e.g., a second client device 100' and / or a third client device 100'') if the video processing server 300 is unable to modify the video stream 190 (e.g., due to insufficient resources or processing limitations). If the video processing server 300 is unable to modify the video stream 190 as requested, the video processing server 300 may receive instructions from the first client device 100 not to forward the video stream 190 to other client devices in the video conferencing session (e.g., a second client device 100' and / or a third client device 100''). If one or more requested visual effects relate to privacy (e.g., the user does not want other users in the video conferencing session to see their unblurred background), this aspect provides further privacy to the user of the first client device 100. Thus, other (remote) client devices in the video conferencing session may receive the modified video stream (with one or more visual effects applied by the video processing server 300 as requested by the first client device 100), or they may not receive any video at all from the first client device 100 during intermittent failures of the video processing server 300.

[0084] Referring to Figure 7, method 700 includes an operation 710 in which a computing device (e.g., the computing device of the first client device 100) determines whether to process the video relating to the video conferencing session in a first mode or a second mode. For example, the first client device 100 may be configured to toggle or switch between a first mode in which the first client device 100 requests the video processing server 300 to apply visual effects to a video stream 190 generated by the first client device 100, and a second mode in which the first client device 100 applies visual effects to the video stream itself and sends the modified video stream (e.g., the input video stream 210 in Figure 2A) to the video processing server 300 (e.g., via the first server 200), and the video processing server 300 then resizes the modified video stream as needed and outputs the resized modified video stream(s) to one or more other client devices participating in the video conferencing session. In some cases, the corrected video stream may be transmitted by the first client device 100 to one or more other client devices participating in the video conferencing session without passing through the video processing server (e.g., via one or more servers, such as the first server 200, the second server 200', or the third server 200''). In some cases, a remote client device requesting the highest resolution video stream may receive the corrected video stream from the first client device 100 via another server (e.g., one of the first server 200, the second server 200', or the third server 200''), and the corrected video stream may not pass through the video processing server 300 (because the video processing server 300 does not need to perform a resizing operation).

[0085] In response to a decision to process the video in a first mode, in operation 720, the method includes a computing device (e.g., a first client device 100) generating a first video stream (e.g., an input video stream 230) and sending it to a server (e.g., a video processing server 300), and in operation 730, the method includes the computing device (e.g., a first client device 100) sending to the server (e.g., a video processing server 300) visual effects information (e.g., visual effects information 230') relating to one or more visual effects to be applied by the video processing server 300 to the first video stream (e.g., an input video stream 230), where the first video stream (e.g., an input video stream 230) corresponds to a raw video stream generated from image data captured by camera 140, which may correspond to a user's self-view. Therefore, in the first mode, the first client device 100 sends a raw video stream without visual effects to the video processing server 300, and further sends visual effect information regarding one or more visual effects to be applied to the first video stream to the video processing server 300.

[0086] In response to a decision to process the video in a second mode, operation 780 includes the computing device (e.g., the first client device 100) generating a second video stream (e.g., input video stream 210) by applying one or more visual effects to the first video stream (before the first video stream is actually transmitted). Operation 790 includes the computing device (e.g., the first client device 100) transmitting the second video stream (e.g., input video stream 230) to a server (e.g., video processing server 300), or transmitting the second video stream to another server (e.g., a separate server system which may include any of the first server 200, the second server 200', the third server 200'', etc.), thus completely bypassing the video processing server 300.

[0087] In some embodiments, a computing device (e.g., a first client device 100) may be configured to operate in mixed mode. For example, the first client device 100 may operate in mixed mode when switching from a first mode to a second mode in which the first client device 100 applies visual effects to a video stream. If, in operation 740, it is determined that the first client device 100 operates in mixed mode (e.g., before switching to the second mode), in operation 760, one or more processors 110 of the first client device 100 are configured to generate a second video stream by having the first client device 100 apply one or more visual effects to the first video stream, and in operation 770, the first client device 100 displays the second video stream as a self-view. Simultaneously, the first client device 100 transmits the first video stream and visual effects information to a server (e.g., a video processing server 300). In other words, while the first client device 100 is in mixed mode, the unmodified video stream and visual effects information continue to be sent to the video processing server 300, which in turn continues to apply one or more visual effects to the unmodified video stream, and one or more modified video streams (with visual effects applied by the video processing server 300) are sent to be displayed on remote client devices participating in the video conferencing session (e.g., the second client device 100' and / or the third client device 100''). When mixed mode is implemented, the switch from the first mode to the second mode is not perceived by the remote client devices at all, however, in the self-view on the first client device 100, a minimal interruption or glitch may occur when switching to mixed mode due to the different timelines of the video streams. In this embodiment, to avoid operating in mixed mode for an extended period, the first client device 100 may be configured to switch from mixed mode to the second mode after a predetermined time (e.g., 5 minutes).When the self-view is determined to be below a threshold size, this method prevents the waste of cloud resources, and thus the first client device 100 can apply visual effects without excessively degrading the self-view.

[0088] If, in operation 740, it is determined that the first client device 100 will not operate in mixed mode, then in operation 750, the first client device 100 receives a modified video stream (e.g., modified video stream 196) from the server (e.g., video processing server 300) and displays the modified video stream on the display 160 as a self-view. For example, the modified video stream may include the first video stream with one or more visual effects applied by the server (e.g., video processing server 300).

[0089] Referring to Figure 8, method 800 includes operation 810 in which a computing device (e.g., the computing device of the first client device 100) determines whether the size of the self-view exceeds a threshold level. For example, the visual effect mode selector 134 may be configured to toggle or switch between a first mode in which the first client device 100 requests the video processing server 300 to apply visual effects to a video stream generated by the first client device 100, based on the size of the self-view and / or the physical resolution of the display of the first client device 100, and a second mode in which the first client device 100 applies visual effects to the video stream and sends the modified video stream to the video processing server 300. The size of the self-view may correspond to the active resolution of the self-view and may be determined by the self-view size identifier 132. For example, during a video conferencing session, if the size of the self-view displayed on the client device's display exceeds a threshold level (value), the operation 820 of the computing device (e.g., the computing device of the first client device 100) includes the visual effects mode selector 134 deciding to switch to (or maintain the second mode). When the size of the self-view exceeds the threshold level (value), a video quality problem perceptible to the user may occur on the client device. Conversely, if the size of the self-view displayed on the display of the first client device 100 is less than the threshold level (value), the operation 830 of the computing device (e.g., the computing device of the first client device 100) includes the visual effects mode selector 134 deciding to switch to (or maintain the first mode).

[0090] In some embodiments, one or more processors 110 are configured to decide to switch to a second mode and / or stop transmitting the video stream to the video processing server 300 in response to the media quality of the corrected video stream (e.g., self-view) being below a threshold level and / or the latency of the corrected video stream received from the video processing server 300 exceeding a threshold level. For example, whether the quality of the corrected video stream (e.g., self-view) is below a threshold can be determined according to various factors and metrics relating to quality of service and quality of experience (e.g., whether the resolution of the self-view is below a threshold level, whether the self-view freezes a certain number of times within a given time, whether the self-view freezes for longer than a given time, the signal-to-noise ratio of the self-view, etc.).

[0091] In some embodiments, in response to the media quality of the modified video stream sent to the first client device 100 being below a threshold level, and / or the latency of the modified video stream sent to the first client device 100 being above a threshold level, the video processing server 300 is configured to send a notification to the first client device 100 indicating that the first client device 100 will take control of applying one or more visual effects to the video stream (switch to a second mode), and / or that the video processing server 300 will stop sending one or more modified video streams to one or more other client devices participating in the video conferencing session (e.g., a second client device 100' and a third client device 100'').

[0092] In one embodiment, when multiple self-views are displayed on the display of the first client device 100 during a video conferencing session, the self-view size identifier 132 is configured to identify the self-view with the largest size. One or more processors 110 of the first client device 100 are configured to determine whether the size of the self-view with the largest size exceeds a threshold level (value) and whether to switch to (or maintain) the second mode. The self-view size identifier 132 may be configured to track or monitor the size of one or more self-views during the video conferencing session to determine whether there is a change in the maximum size and whether to switch to (or maintain) the second mode. Changes in the size and / or position of a self-view may be sent to the video processing server 300 to update the position of visual effects applied to the self-view when the first client device is in the first mode. In the embodiment, when the size of the self-view is frequently changed by the user (i.e., changed more than a predetermined number of times within a predetermined time, such as four or more times in 10 seconds), the self-view size identifier 132 may be configured to wait until the size changes are stopped for a predetermined time (e.g., 3 seconds) before determining the size of the self-view. One or more processors 110 may then determine whether the size of the self-view displayed on the display of the first client device 100 exceeds a threshold level (value).

[0093] Referring to Figure 9, method 900 includes operation 910 in which a computing device (e.g., the computing device of the first client device 100) determines whether a zoom function (e.g., auto-zoom or centering function) is enabled in the video conferencing application on the first client device 100. That is, when one or more processors 110 determine that a zoom function is enabled, operation 920 may include the visual effects mode selector 134 switching to a second mode or maintaining the second mode. For example, applying visual effects to resize an enlarged video stream may have a negative impact on the corrected video stream, such as a decrease in quality (e.g., resolution), so the video processing server 300 may not be permitted to apply visual effects to the enlarged video stream in the first mode. When one or more processors 110 determine that a zoom function is not enabled, operation 930 may include the visual effects mode selector 134 allowing the first client device 100 to switch to a first mode or maintaining the first mode.

[0094] In an alternative embodiment, when the zoom function is enabled in the video conferencing application on the first client device 100, the video processing server 300 may be configured to apply visual effects to the video stream in a first mode. For example, an enlarged stream (i.e., the raw video stream modified by zoom on the first client device 100) may be sent from the first client device 100 to the video processing server 300, and visual effects may be applied to the enlarged stream. For example, a raw video stream may be sent from the first client device 100 to the video processing server 300, and the video processing server 300 may apply both enlargement and visual effects to the video stream.

[0095] Referring to Figure 10, Method 1000 includes an operation 1010 of a computing device (e.g., the computing device of the first client device 100) in which one or more processors 110 determine whether the user of the first client device 100 is in the waiting room of a video conferencing session or has joined a video conferencing session. For example, when switching between a first mode and a second mode, the first client device 100 may be configured to distinguish between a “waiting room” state, where the user is waiting to be allowed to join or join a video conferencing session meeting, and a “joined” state, where the user has been allowed to join or is joining a meeting. For example, in some embodiments, when the first client device 100 is in a waiting state, the video processing server 300 cannot support sending or receiving media to the first client device 100 while the first client device 100 remains in a waiting state. For example, in operation 1020, the visual effects mode selector 134 may be configured to switch from a second mode to a first mode, or to maintain the first mode, in response to one or more processors 110 determining that the first client device 100 has transitioned from a waiting state to a joined state. For example, in operation 1030, the visual effects mode selector 134 may be configured to switch from a first mode to a second mode, or to maintain the second mode, in response to one or more processors 110 determining that the first client device 100 is in a waiting state. For example, while the first client device 100 is waiting in the waiting state to be allowed to join the meeting, any background blur effect (or other visual effect) applied to the video stream by the first client device 100 is "transitioned" from the first client device 100 to the video processing server 300 in response to the first client device 100 being allowed to join the video conference session meeting. Therefore, when the first client device 100 joins the meeting, the video processing server 300 is responsible for applying a background blur effect (or other visual effects) to the video stream.In an alternative embodiment, the video processing server 300 may support sending and receiving media to and from the first client device 100 while the first client device 100 is in a standby state, and therefore the operation shown in Figure 10 cannot be performed.

[0096] When the first client device 100 toggles or switches between the first mode and the second mode, intermittent video problems (e.g., video freezing, frame drops, etc.) may occur. Such intermittent video problems may be more apparent in the self-view than in the view by remote participants in a video conferencing session. The exemplary embodiments described below provide additional methods for avoiding or mitigating these video problems that occur when switching between the first mode and the second mode. For example, when switching from the first mode to the second mode (or vice versa), the first client device 100 may be configured to wait until the second mode (or first mode) has started before stopping the previous embodiment. For example, when switching from the second mode to the first mode, the first client device 100 begins receiving remote frames of the self-view generated by the video processing server 300 before the first client device 100 stops applying visual effects to the video stream it sends to the video processing server 300. As a result, some frames of the video stream(s) received by the first client device 100 and remote client devices (e.g., the second client device 100' and the third client device 100'') may have double the visual effects applied to the user's self-view on the first client device 100. For example, when switching from the first mode to the second mode, before the video processing server 300 stops applying visual effects to the video stream sent to the video processing server 300 and looped back to the first client device 100, the first client device 100 begins generating and displaying frames of the self-view with visual effects applied by the first client device 100 to the video stream. Again, some frames of the video stream(s) received by the first client device 100 and remote client devices may have double the visual effects applied to the user's self-view on the first client device 100.

[0097] For example, if the first client device 100 is applying a background replacement visual effect to a video stream when the video processing server 300 switches to a first mode in which it applies background blur to the video stream, the visual effect mode selector 134 of the first client device 100 may be configured to temporarily operate in hybrid or mixed mode, thereby allowing the video processing server 300 to apply background blur to the video stream while simultaneously allowing the first client device 100 to still apply the background replacement visual effect to the video stream for a predetermined time (or until the first keyframe of the remotely modified video stream generated by the video processing server 300 applying background blur to the video stream is received by the first client device 100). After the predetermined time has elapsed (or after the first keyframe has been received), the visual effect mode selector 134 is configured to disable the background replacement visual effect that the first client device 100 is applying, and thus the first client device 100 operates in the first mode. Therefore, while some frames of the video stream may contain both visual effects for a short time, this technique ensures that no unprocessed frames are transmitted to other remote client devices participating in the video conferencing session (e.g., a second client device 100' and a third client device 100'').

[0098] In an exemplary embodiment, when switching from a second mode to a first mode, one or more processors 110 of the first client device 100 may be configured to continue displaying the user's self-view of the first client device 100 via the local stream until a first keyframe of the remotely modified video stream is received from the video processing server 300. As a result of this technique, freezes are eliminated and the number of frames with double effects applied (e.g., double-blurred frames with visual effects applied by both the first client device 100 and the video processing server 300) can be minimized. If the video stream has not yet been resized to accommodate multiple resolutions, the remote views on other client devices during the video conferencing session (e.g., the second client device 100' and the third client device 100'') may temporarily freeze when the first client device 100 requests the video processing server 300 to apply visual effects, and may recover from the freeze once the first keyframe is propagated from the first client device 100 to the other client devices via the video processing server 300. The first client device 100 continues to apply the visual effects until it receives a new modified video stream from the video processing server 300. Thus, the first client device 100 receives several frames to which the effects are applied twice. In another embodiment, to make the switch from the second mode to the first mode in SelfView faster, SelfView view requests may be limited to a threshold resolution (e.g., 640x360).

[0099] For example, when switching from the first mode to the second mode, one or more processors 110 of the first client device 100 may be configured to immediately display a self-view using the local stream as soon as the first client device 100 applies a visual effect, without any artifacts being perceived. Remote views on other client devices during the video conferencing session (e.g., second client device 100' and third client device 100'') may display some frames with double effects applied (e.g., double blur) until the video processing server 300 disables the visual effect, and may display some stalls / frame drops while the first client device 100 is loading the visual effect.

[0100] In some embodiments, other configurations may be implemented in the first client device 100 and / or the video processing server 300 to reduce freeze time and / or reduce or avoid double-effect frames. For example, the video processing server 300 may be kept in a state ready to apply visual effects to the video stream transmitted from the first client device 100 (e.g., ready state), but does not actually apply any visual effects until the mode is switched to the first mode. By keeping the video processing server 300 ready to apply visual effects, the startup time of the video processing server 300 is reduced, and the freeze time of remote client devices (e.g., second client device 100' and third client device 100'') during the video conferencing session is reduced. In embodiments, the first client device 100 may be configured to send a timestamp / frame counter to the video processing server 300 indicating a future time when the first client device 100 wants to switch modes. The video processing server 300 may be configured to ensure that the visual effects are enabled / disabled in that frame at the appropriate time. The first client device 100 may be configured to wait until a timestamp / frame counter value is reached before starting / stopping the application of the visual effect, thereby blocking the frames of the double effect. According to this embodiment, where the first client device 100 sends the timestamp / frame counter, the video processing server 300 may be configured to switch modes immediately and / or remain in a ready state.

[0101] In this embodiment, the first client device 100 may be configured to transmit metadata for frames of the video stream, which provides the video processing server 300 with information about visual effects. When the first client device 100 requests the video processing server 300 to apply visual effects, the video processing server 300 is configured to generate a modified video stream with visual effects only if the visual effects to be applied are different from any of the visual effects applied to the input video stream. According to this embodiment, where the first client device 100 transmits metadata for frames of the video stream that provides the video processing server 300 with information about visual effects, the video processing server 300 may be configured to switch modes immediately and / or remain in a ready state.

[0102] For example, in some embodiments, the first client device 100 and / or the video processing server 300 may preload visual effects without actually applying them to the output stream. For example, when switching from a first mode to a second mode, the first client device 100 may be configured to preload visual effects, request the video processing server 300 to freeze the current frame to which the visual effects have been applied by the video processing server 300, transmit the video stream to which the visual effects have been applied by the first client device 100, and then request the video processing server 300 to stop applying the visual effects to the video stream. Thus, instead of a double-effect frame, a short video freeze is added. For example, when switching from a second mode to a first mode, the video processing server 300 may be configured to receive a request from the first client device 100 to preload visual effects and freeze the frame to which the visual effects have been applied by the first client device 100. The first client device 100 may stop applying visual effects to the video stream and send the video stream without visual effects to the video processing server 300. The video processing server 300 is then requested to apply visual effects to the video stream. Thus, a short video freeze is added instead of a double-effect frame.

[0103] In some embodiments, the video conferencing application 130 may have a default setting for the first client device 100 to operate in a first mode. That is, the video conferencing application 130 may, by default, prefer that the video processing server 300 apply visual effects in a first mode during a video conferencing session (rather than the first client device 100 applying visual effects to the video stream in a second mode). In response to the video processing server 300 being unavailable (e.g., due to insufficient resources, high peak usage, bandwidth limitations, etc.), the visual effects mode selector 134 may be configured to switch to a second mode, thereby allowing the processing of visual effects to be applied by the first client device 100. For example, in response to insufficient graphics computing resources on the video processing server 300, the video processing server 300 notifies the first client device 100 (e.g., via the first server 200) via a message and metadata that the first mode is unavailable, and the visual effect mode selector 134 switches the first client device 100 to a second mode in which visual effects (i.e., client-side effects) are applied. In an exemplary embodiment, if the first client device 100 cannot support a visual effect requested to be applied to the video stream, one or more processors 110 of the first client device 100 may be configured to control the display 160 of the first client device 100 to display a message indicating that the requested visual effect cannot be applied to the video stream, and the camera 140 of the first client device 100 may be muted to protect user privacy. Additionally or alternatively, if the first client device 100 cannot support a visual effect requested to be applied to the video stream, one or more processors 110 of the first client device 100 may be configured to stop transmitting the video stream. The first client device 100 may also notify the user, in an additional or alternative manner (for example, via a speaker), that the requested visual effect cannot be applied to the video stream.

[0104] In some embodiments, the first client device 100 may be configured to display a user self-view of the first client device 100 during a video conferencing session, the self-view may be generated and displayed locally by the first client device 100, or the self-view may correspond to a video stream modified by a video processing server 300 applying one or more visual effects to the video stream, and the modified video stream is looped back to the first client device 100 and displayed as the user's self-view. For example, if the first client device 100 operates in a first mode and requests the video processing server 300 to apply visual effects to the video stream, the first client device 100 may be configured to receive the modified video stream "remotely". The modified video stream received remotely may have worse quality and latency compared to the local self-view generated by the first client device 100 due to network round trips and transcoding performed by the video processing server 300. However, according to the exemplary embodiments disclosed herein, the quality and latency of the self-view can be improved according to various techniques. Thus, the media quality and latency of the self-view can be maintained at a level that does not adversely affect the user's perception of the quality of the video conferencing session. For example, to obtain lower latency (e.g., a threshold of less than 1 second, a threshold of less than 300 ms, etc.), the video processing server 300 may be configured to enable WebRTC low-latency rendering and set latency threshold limits when sending the corrected video stream back to the first client device 100 (e.g., by setting threshold limits in the PlayoutDelayLimits RTP header extension).As an additional or alternative technique to improve self-view quality, the video processing server 300 may be configured to adjust its bitrate allocation strategy to prioritize sending the corrected video stream back to the first client device 100 rather than sending it to other client devices (e.g., a second client device 100' and a third client device 100''). That is, when distributing available bandwidth downstream, the video processing server 300 may be configured to prioritize the corrected video stream 196 to be looped back. If this prioritization is not performed, in larger meetings, the user of the first client device 100 will lose their self-view when not speaking, while other users in the video conference session can still see the user. As an additional or alternative technique to improve self-view quality, one or more processors 110 of the first client device 100 may be configured, as a performance adaptation, to reduce the frame rate of remotely received video streams (e.g., video streams sent from other client devices during the video conferencing session, such as the second client device 100' and the third client device 100''), but the modified video stream for self-view will be excluded (i.e., maintained at the default frame rate).If, despite one or more of the aforementioned techniques being implemented (for example, by prioritizing the modified video stream sent back to the first client device 100), there is still insufficient bandwidth for the modified video stream 196 to be looped back, one or more processors 110 of the first client device 100 may be configured to detect that the video processing server 300 and / or the first server 200 cannot modify the video stream 190 generated by the first client device 100 with visual effects, the visual effects mode selector 134 may switch from processing the video in a first mode to processing the video in a second mode, and the first client device 100 modifies the video stream by applying visual effects to the video stream and generates its own self-view.

[0105] Terms such as “module” and “unit” may be used herein in connection with various features of this disclosure. Such terms may refer to, but are not limited to, software or hardware components or devices that perform a particular task, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). A module or unit may be configured to reside on an addressable storage medium and to run on one or more processors. Thus, a module or unit may include, as an example, software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided by components and modules / units may be combined into fewer components and modules / units, or further separated into additional components and modules.

[0106] The embodiments of the exemplary models described above may be recorded on a computer-readable medium (e.g., a non-temporary computer-readable medium) containing program instructions for performing various operations embodied by a computer. The medium may also contain program instructions, data files, and data structures, either alone or in combination. Examples of non-temporary computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs, Blu-ray discs, and DVDs; magneto-optical media such as optical discs; and other hardware devices specifically configured to store and execute program instructions, such as semiconductor memory, read-only memory (ROM), random access memory (RAM), flash memory, and USB memory. Examples of program instructions include both machine code, such as that generated by a compiler, and files containing higher-level code that can be executed by a computer using an interpreter. The program instructions may be executed by one or more processors. To perform the operations of the embodiments described above, the hardware devices described may be configured to function as one or more software modules, and vice versa. Furthermore, non-temporary computer-readable storage media can be distributed across computer systems connected via a network, and computer-readable code or program instructions can be stored and executed in a non-centralized manner. Additionally, non-temporary computer-readable storage media can also be embodied in at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA).

[0107] Each block in a flowchart diagram may represent a unit, module, segment, or portion of code containing one or more executable instructions for performing a specific logical function. Note that in some alternative embodiments, the functions described in a block may be executed out of order. For example, two consecutively shown blocks may actually be executed nearly concurrently (simultaneously), or blocks may sometimes be executed in reverse order depending on the functions involved.

[0108] While this disclosure has described various exemplary embodiments, each example is provided for illustrative purposes only and is not intended to limit the disclosure. Those skilled in the art, upon understanding the foregoing, will readily be able to create variations, modifications, and equivalents of such embodiments. Therefore, this disclosure does not preclude the inclusion of such modifications, modifications, and / or additions to the disclosed subject matter, as will be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used in conjunction with another embodiment to create yet another embodiment. Thus, this disclosure is intended to cover such variations, modifications, and equivalents.

Claims

1. One or more memory locations configured to store instructions, One or more processors configured to execute the instructions stored in one or more of the memory, A server system comprising the above, wherein one or more processors execute the above instructions, Receiving a video stream related to a video conference session from the first client device, Receiving visual effect information relating to one or more visual effects to be applied to the video stream from the first client device, Based on the received visual effect information, apply one or more visual effects to the video stream to generate one or more modified video streams, Sending the one or more modified video streams to one or more other client devices participating in the video conference session, Sending one of the one or more corrected video streams to the client device described above, The server system performs the following actions: in response to the media quality of the modified video stream transmitted to the first client device being below a threshold level, and / or the latency of the modified video stream transmitted to the first client device being above a threshold level, the first client device will be responsible for controlling the application of the one or more visual effects to the video stream, and / or one or more other client devices participating in the video conferencing session will be notified to stop transmitting the one or more modified video streams.

2. The one or more processors described above are: Based on the received visual effects information, generate multiple modified video streams, each having a different resolution. To each of the one or more other client devices participating in the video conferencing session, one of the multiple modified video streams is transmitted based on the data transfer rate between the server system and each client device. The server system according to claim 1, configured to perform the following:

3. The server system according to claim 1, wherein the visual effects information includes information relating to one or more settings of the one or more visual effects applied to the video stream.

4. The server system according to claim 3, wherein one or more of the settings include a blur radius for a background blur effect applied to at least a portion of the video stream.

5. The server system according to claim 3, wherein one or more of the settings include a brightness level for a light-and-dark effect applied to at least a portion of the video stream.

6. One or more memories configured to store instructions, One or more processors configured to execute the instructions stored in one or more of the memory, A server system comprising the above, wherein one or more processors execute the above instructions, Receiving a video stream related to a video conference session from the first client device, Receiving visual effect information relating to one or more visual effects to be applied to the video stream from the first client device, Based on the received visual effect information, apply one or more visual effects to the video stream to generate one or more modified video streams, The process involves sending the one or more modified video streams to one or more other client devices participating in the video conference session, and performing the following: The video stream transmitted from the first client device is a self-view of the user of the first client device. In response to the user's self-view being changed to a size exceeding a threshold level, one or more processors: To stop applying one or more of the aforementioned visual effects to the video stream, Sending a notification to the first client device indicating that the first client device is responsible for controlling the application of the one or more visual effects to the video stream, and / or to the one or more other client devices participating in the video conference session to stop transmitting the one or more modified video streams, A server system configured to execute [the specified command / function].

7. One or more memory locations configured to store instructions, One or more processors configured to execute the instructions stored in one or more of the memory, A computing device comprising, wherein one or more processors execute the instructions, To decide whether to process the video related to the video conference session in the first mode or the second mode, In response to the decision to process the video in the first mode, a first video stream is generated, the first video stream is transmitted to the server system, and visual effect information relating to one or more visual effects to be applied to the first video stream is transmitted to the server system. In response to the decision to process the video in the second mode, a second video stream is generated by applying one or more visual effects to the first video stream, and the second video stream is transmitted to the server system or another server system. Receiving from the server system a modified video stream in which one or more visual effects have been applied to the first video stream based on the visual effect information transmitted to the server system, The computing device determines and performs the following actions in response to the media quality of the corrected video stream being below a threshold level and / or the latency of the corrected video stream received from the server system exceeding a threshold level: to process the video in the second mode and / or to stop transmitting the first video stream to the server system.

8. The computing device according to claim 7, wherein the visual effects information includes information regarding one or more settings of visual effects to be applied to the first video stream.

9. The computing device according to claim 8, wherein one or more of the settings include a blur radius for a background blur effect applied to at least a portion of the first video stream.

10. The computing device according to claim 8, wherein the one or more settings include a brightness level of a light-and-dark effect to be applied to at least a portion of the first video stream.

11. One or more memories configured to store instructions, One or more processors configured to execute the instructions stored in one or more of the memory, A computing device comprising, wherein one or more processors execute the instructions, To decide whether to process the video related to the video conference session in the first mode or the second mode, In response to the decision to process the video in the first mode, a first video stream is generated, the first video stream is transmitted to the server system, and visual effect information relating to one or more visual effects to be applied to the first video stream is transmitted to the server system. In response to the decision to process the video in the second mode, the system generates a second video stream by applying one or more visual effects to the first video stream, and transmits the second video stream to the server system or another server system. The video includes a self-view of the user of the computing device, The one or more processors are configured to determine whether to process the video in the first mode or the second mode based on the size of the self-view. When the size of the self-view is less than a threshold level, one or more processors are configured to decide to process the video in the first mode. A computing device in which, when the size of the self-view exceeds a threshold level, one or more processors are configured to decide to process the video in the second mode.

12. One or more memories configured to store instructions, One or more processors configured to execute the instructions stored in one or more of the memory, A computing device comprising, wherein one or more processors execute the instructions, To decide whether to process the video related to the video conference session in the first mode or the second mode, In response to the decision to process the video in the first mode, a first video stream is generated, the first video stream is transmitted to the server system, and visual effect information relating to one or more visual effects to be applied to the first video stream is transmitted to the server system. In response to the decision to process the video in the second mode, the system generates a second video stream by applying one or more visual effects to the first video stream, and transmits the second video stream to the server system or another server system. The video includes multiple self-views of the user of the computing device, The one or more processors are configured to identify the self-view having the largest size from among the plurality of self-views. A computing device in which one or more processors are configured to determine whether to process the video in a first mode or a second mode based on the size of the self-view having the maximum size.

13. The computing device according to claim 7, wherein the visual effects information includes a command that the server system does not transmit the first video stream to other client devices participating in the video conferencing session if the one or more visual effects cannot be applied to the first video stream.

14. To decide whether to process the video related to the video conference session in the first mode or the second mode, In response to the decision to process the video in the first mode, a first video stream is generated, the first video stream is transmitted to the server system, and visual effect information relating to one or more visual effects to be applied to the first video stream is transmitted to the server system. In response to the decision to process the video in the second mode, a second video stream is generated by applying one or more visual effects to the first video stream, and the second video stream is transmitted to the server system or another server system. Receiving from the server system a modified video stream in which one or more visual effects have been applied to the first video stream based on the visual effect information transmitted to the server system, A computer implementation method comprising determining to process the video in the second mode and / or to stop transmitting the first video stream to the server system in response to the media quality of the modified video stream being below a threshold level and / or the latency of the modified video stream received from the server system exceeding a threshold level.

Images