Cloud-based visual effect application to video

Abstract

A server system (300) receives, from a first client device (100), a video stream (190, 190 ') related to a video conference session, and receives, from the first client device, visual effect information related to one or more visual effects to be applied to the video stream. The server system applies the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams (192, 194) and transmits the one or more modified video streams to one or more other client devices participating in the video conference session.

Description

Priority claim

[0001] This application is based on and claims priority to U.S. application No. 17 / 738,176, filed on May 6, 2022, the entire text of which is incorporated herein by reference. Technical Field

[0002] The present disclosure relates to video processing for video telephony (which may also be referred to as video calling or video conferencing) conducted over a network. More specifically, embodiments disclosed herein relate to video conferencing where video processing operations are performed by one or more servers in the cloud rather than by a client device. Background Art

[0003] Video telephony comprises a telecommunication system in which a computing device is used to transmit audio and visual signals, thereby allowing two or more persons from different locations to participate in a real-time conversation.

[0004] Some video phone applications support various visual effects, such as blurred backgrounds and adjusted lighting, however, not all client devices (e.g., laptops, tablets, smartphones, etc.) are powerful enough to render these visual effects without degrading the performance of the client device and / or the video conferencing application. Summary of the invention

[0005] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned by practicing example embodiments.

[0006] In one or more example embodiments, a server system (e.g., a video processing server) includes one or more memories configured to store instructions, and one or more processors configured to execute the instructions stored in the one or more memories to: receive a video stream associated with a video conferencing session from a first client device, receive visual effect information associated with one or more visual effects to be applied to the video stream from the first client device, apply the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams, and transmit the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0007] In some implementations, the one or more processors of the server system are configured to transmit a modified video stream of the one or more modified video streams to the first client device.

[0008] In some implementations, one or more processors of the server system are configured to, in response to a media quality of a modified video stream transmitted to a first client device being less than a threshold level and / or a latency of the modified video stream transmitted to the first client device being greater than a threshold level, transmit a notification to the first client device indicating that the first client device is to take over control to apply one or more visual effects to the video stream and / or cease transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0009] In some implementations, one or more processors of a server system are configured to: generate multiple modified video streams based on the received visual effect information, wherein each of the multiple modified video streams has a different resolution; and transmit one of the multiple modified video streams to a corresponding client device based on a data transmission rate between the server system and the corresponding client device among one or more other client devices participating in the video conferencing session.

[0010] In some implementations, the visual effect information includes information about one or more settings of one or more visual effects to be applied to the video stream. In some implementations, the one or more settings include a blur radius of a background blur effect to be applied to at least a portion of the video stream. In some implementations, the one or more settings include a brightness level of a lighting effect to be applied to at least a portion of the video stream.

[0011] In some implementations, the video stream transmitted from the first client device is a self-view of a user of the first client device. In some implementations, in response to the user's self-view being changed to a size greater than a threshold level, the one or more processors of the server system are configured to: stop applying the one or more visual effects to the video stream, and transmit a notification to the first client device indicating that the first client device will take over control to apply the one or more visual effects to the video stream and / or stop transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0012] In one or more example embodiments, a computing device (e.g., a client device) includes one or more memories configured to store instructions, and one or more processors configured to execute the instructions stored in the one or more memories to: determine whether to process video associated with a video conferencing session in a first mode or a second mode, in response to determining to process the video in the first mode, generate a first video stream and transmit the first video stream to a server system, and transmit visual effect information related to one or more visual effects to be applied to the first video stream to the server system, and in response to determining to process the video in the second mode, generate a second video stream by applying the one or more visual effects to the first video stream, and transmit the second video stream to the server system or a separate server system.

[0013] In some implementations, one or more processors of a computing device are configured to receive, from a server system, a modified video stream in which one or more visual effects have been applied to a first video stream based on visual effects information transmitted to the server system.

[0014] In some implementations, one or more processors of the computing device are configured to determine to process the video in a second mode and / or to stop transmitting the first video stream to the server system in response to a media quality of the modified video stream being less than a threshold level and / or a latency of the modified video stream received from the server system being greater than a threshold level.

[0015] In some implementations, the visual effect information includes information about one or more settings of a visual effect to be applied to the first video stream. In some implementations, the one or more settings include a blur radius of a background blur effect to be applied to at least a portion of the first video stream. In some implementations, the one or more settings include a brightness level of a lighting effect to be applied to at least a portion of the first video stream.

[0016] In some implementations, the video includes a self-view of a user of the computing device, and the one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on a size of the self-view. In some implementations, when the size of the self-view is less than a threshold level, the one or more processors of the computing device are configured to determine to process the video in the first mode, and when the size of the self-view is greater than the threshold level, the one or more processors of the computing device are configured to determine to process the video in the second mode.

[0017] In some implementations, a video includes multiple self-views of a user of a computing device, one or more processors of the computing device are configured to determine a self-view having a largest size among the multiple self-views, and the one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on the size of the self-view having the largest size.

[0018] In some implementations, the visual effect information includes instructions for the server system to not transmit the first video stream to other client devices participating in the video conferencing session if one or more visual effects are not applicable to the first video stream.

[0019] In one or more example embodiments, a computer-implemented method for a computing device (e.g., a client device) includes determining whether to process video associated with a video conferencing session in a first mode or a second mode, in response to determining to process the video in the first mode, generating a first video stream and transmitting the first video stream to a server system, and transmitting visual effect information associated with one or more visual effects to be applied to the first video stream to the server system, and in response to determining to process the video in the second mode, generating a second video stream by applying the one or more visual effects to the first video stream, and transmitting the second video stream to the server system or a separate server system.

[0020] In one or more example embodiments, a computer-implemented method for a server system (e.g., a video processing server) includes receiving a video stream associated with a video conferencing session from a first client device, receiving visual effect information associated with one or more visual effects to be applied to the video stream from the first client device, applying the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams, and transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0021] In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions executable by one or more processors of a client device and / or a server system. In some implementations, the computer-readable medium stores instructions that may include instructions to cause one or more processors to perform one or more operations (e.g., operations of a server system and / or operations of a client device) associated with any of the methods described herein. As described herein, the computer-readable medium may store additional instructions to perform other aspects of the server system and client device and corresponding methods of operation.

[0022] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description, drawings and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] A detailed discussion of example embodiments for persons of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0024] Figure 1 An example video conferencing system according to one or more example embodiments of the present disclosure is depicted;

[0025] FIG. 2A to FIG. 2B An example flow chart of a video processing server according to one or more example embodiments of the present disclosure is depicted;

[0026] Figure 3 depicts an example block diagram of a client device and a video processing server according to one or more example embodiments of the present disclosure;

[0027] Figure 4 depicts an example display of a video conferencing session at a client device according to one or more example embodiments of the present disclosure;

[0028] Figure 5 depicts an example display of visual effects applied to a user's self-view in a video conferencing session at a client device; and

[0029] Figures 6 to 10 A flowchart of an example non-limiting computer-implemented method is shown according to one or more example embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Reference will now be made to embodiments of the present disclosure, one or more examples of which are illustrated in the accompanying drawings, wherein like reference numerals represent like elements. Each example is provided by way of explanation of the present disclosure and is not intended to limit the present disclosure. In fact, it will be apparent to those skilled in the art that various modifications and changes may be made to the present disclosure without departing from the scope or spirit of the present disclosure. For example, a feature illustrated or described as part of an embodiment may be used together with another embodiment to produce yet further embodiments. Therefore, the present disclosure is intended to encompass modifications and changes within the scope of the appended claims and their equivalents.

[0031] The terms used herein are used to describe example embodiments and are not intended to limit and / or constrain the present disclosure. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. In the present disclosure, terms such as "including", "having", "comprising", etc. are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0032] It will be understood that although the terms first, second, third, etc. can be used in this article to describe various elements, these elements should not be limited to these terms. On the contrary, these terms are used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, a first element can be referred to as a second element, and a second element can be referred to as a first element.

[0033] It will be understood that when an element is referred to as being “connected to” another element, the expression encompasses examples of being directly connected or directly coupled as well as being connected or coupled with one or more other elements interposed therebetween.

[0034] The term "and / or" includes a combination of multiple related listed items or any of the multiple related listed items. For example, the scope of the expression or phrase "A and / or B" includes the item "A", the item "B" and the combination of the items "A and B".

[0035] In addition, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A; (2) at least one of B; and (3) at least one of A and at least one of B. Likewise, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A; (2) at least one of B; (3) at least one of C; (4) at least one of A and at least one of B; (5) at least one of A and at least one of C; (6) at least one of B and at least one of C; and (7) at least one of A, at least one of B, and at least one of C.

[0036] According to example embodiments, aspects of video processing for a video conference may be performed by one or more servers in the cloud rather than by a client device, thereby alleviating processing tasks (e.g., central processing unit / graphics processing unit (CPU / GPU) tasks) on the client device that may cause the client device to overheat, display poor quality, or otherwise perform poorly. According to example embodiments, one or more servers receive a video stream related to a video conferencing session from a client device. The one or more servers also receive visual effect information related to one or more visual effects to be applied to the video stream from the client device. The one or more servers apply the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams, and transmit the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0037] According to an example of the present disclosure, a video stream is obtained from a client device, and media features corresponding to visual effects to be added to the video stream are processed in the cloud by one or more servers, which are hereinafter referred to as video processing servers. A video processing server is a server (or server combination) configured to receive a video stream from a client device (possibly through an intermediate server) and transmit the video stream (possibly in a modified form) to one or more other client devices. In an example embodiment, the video processing server can modify the video stream by adding one or more visual effects to the video stream based on information received from the client device about one or more visual effects to be applied to the video stream. The video processing server can also change the resolution of the modified or unmodified video stream based on the requirements of the corresponding client device to receive the modified or unmodified video stream. Therefore, the video processing server has a video stream from a client device as an input, and has multiple outputs corresponding to multiple modified or unmodified video streams with different resolutions according to the requirements of the corresponding receiving client device. In an example embodiment, when the video stream is modified by the video processing server, the modified video stream is transmitted to the client device that transmits the video stream to the cloud, and is also transmitted to other client devices participating in the video conference. Thus, a user of the client device may also be able to view a modified video stream having visual effects applied to the video stream during a video conference.

[0038] According to an example embodiment, a client device determines whether to process a video related to a video conferencing session in a first mode or a second mode. In response to determining to process the video in the first mode, the client device generates a first video stream and transmits the first video stream to a server, and transmits visual effect information related to a visual effect to be applied to the first video stream to the server. In response to determining to process the video in the second mode, the client device generates a second video stream by applying a visual effect to the first video stream, and transmits the second video stream to the server (or another server). For example, in the first mode, the client device transmits an original video stream without visual effects to the server, and additionally transmits visual effect information related to the visual effect to be applied to the first video stream. For example, the visual effect can be a blur effect to be applied to the background of the self-view video stream. The server receives the original video stream and applies the blur effect to the background of the self-view video stream based on the visual effect information to generate a modified video stream. The server transmits the modified video stream to other client devices participating in the video conferencing session, and in some cases, additionally transmits the modified video stream to the client device that transmits the original video stream to the server.

[0039] According to the embodiments disclosed herein, visual effects can be applied to video streams by one or more servers, as described above, these servers are referred to as video processing servers. For example, a video processing server can receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to uncompressed media. In order to implement visual effects to a video stream, a video processing server can utilize a library that supports combining visual rendering effects (e.g., blur, zoom, lighting, etc.) with the execution of foreground segmentation (e.g., in the self-view of a user of a client device, segmenting out a user and applying visual effects to a background). A video processing server can be specifically configured to perform video processing on a video stream received from a client device to apply visual effects to a video stream. For example, a video processing server can include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or applying visual effects to a video stream) and multiple video encoder accelerators (e.g., application specific integrated circuits (ASICs) for performing transcoding operations, such as encoding, decoding, and resizing).

[0040] According to the embodiments disclosed herein, a loopback stream from a video processing server to a client device is implemented so that a user of the client device can view visual effects applied to a video stream. In an example embodiment, one or more intermediate servers may be placed between the client device and the video processing server so that the one or more intermediate servers route the video stream from the client device to the video processing server, and the one or more intermediate servers route the modified video stream from the video processing server to the client device.

[0041] The video processing server is configured to perform video transcoding operations. For example, the video processing server can receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to the uncompressed media. To implement visual effects to a video stream, the video processing server can utilize a library that supports combining visual rendering effects (e.g., blurring, scaling, lighting, etc.) with the performance of foreground segmentation (e.g., in a user's self-view of a client device, segmenting out a user and applying visual effects to a background).

[0042] For example, when a client device applies visual effects to the video stream itself, the video processing server can be bypassed and other servers in the computing network (server system) can be used to transmit the video stream to the client devices participating in the video conferencing session. In another example embodiment, when the video processing server is not requested or required to apply visual effects to the incoming video stream, the video processing server is configured to decode the video stream received from the client device and pass the decoded video stream to a plurality of resizer nodes, each resizer node for each unique resolution requested by the corresponding client device to receive the video stream for the video conferencing session. That is, the video processing server is configured to perform video transcoding operations using a node graph. In some implementations where the video processing server does not apply visual effects, a client device requesting a video stream with the highest resolution can receive the video stream from the client device through another server without the video stream passing through the video processing server because no resizing operation is required to be performed by the video processing server. According to another example, the video processing server is configured to pass the decoded video stream to a plurality of resizer nodes, wherein each of the resizer nodes is bucketed to correspond to a resolution at a specific bit rate for a corresponding client device participating in the video conferencing session, which may or may not correspond exactly to the resolution of the corresponding client device. For example, the corresponding client device may have the ability to process a high-definition (HD) video stream (e.g., 1920x1080), but may be limited to a specific resolution (e.g., 800x600) due to bit rate and / or bandwidth limitations, and thus the video processing server may resize the modified video stream according to the specific resolution based on the bit rate of the corresponding client device.

[0043] According to the example embodiments disclosed herein, the client device is configured to request the video processing server to apply one or more visual effects to a video stream generated by the client device and transmitted to the video processing server (e.g., through an intermediate server). The client device can be configured to, for example, signal the video processing server with a configuration property associated with the video stream (e.g., through metadata) to apply the visual effect to the video stream. The configuration property can include one or more settings of one or more visual effects to be applied to the video stream by the video processing server. When the client device requests the video processing server to apply the visual effect to the video stream, the video processing server is configured to decode the video stream received from the client device, and pass the decoded video stream to a visual effect applier (also referred to as a media pipeline effect module) to apply the visual effect to the video stream to generate a modified video stream. The modified video stream is then passed to a plurality of resizer nodes, each resizer node for each unique resolution requested by the corresponding client device to receive the modified video stream. According to another example, a video processing server is configured to pass a modified video stream to a plurality of resizer nodes, wherein each of the resizer nodes is bucketed to correspond to a resolution at a specific bit rate for a corresponding client device participating in a video conferencing session, which may or may not correspond exactly to a resolution of the corresponding client device.

[0044] According to example embodiments disclosed herein, a client device may be configured to receive a modified video stream from a video processing server. The client device may specifically request the video processing server to transmit the modified video stream back to the client device, or the video processing server may be configured to automatically transmit the modified video stream back to the client device. For example, when a client device requests a visual effect, the video processing server is configured to signal (e.g., by a message or by metadata) to an intermediate server (i.e., a server that is placed between the client device and the video processing server and forwards the video stream received from the client device to the video processing server): the video stream received from the client device will also be used at least in part for transmission from the video processing server to the client device through the intermediate server. Therefore, in response to receiving a signal from the video processing server, the intermediate server prepares to send at least a portion of the video stream back to the client device, and sends a push notification to the client device to notify the client device about the new stream. When the client device receives a notification about the modified stream, the client device is configured to perform a search of incoming data from the intermediate server including the modified video stream. The client device may be configured to preferentially display the modified video stream rather than the local stream generated at the client device.

[0045] According to example embodiments disclosed herein, a client device may be configured to toggle or switch between a first mode in which the client device requests a video processing server to apply a visual effect to a video stream generated at the client device and a second mode in which the client device applies the visual effect to the video stream and transmits the modified video stream to the video processing server. For example, if a user of a client device switches to a first mode in which background blur is applied to the video stream by the video processing server while a background replacement visual effect is applied to the video stream, one or more processors of the client device may be configured to first enable the first mode so that the video processing server applies the background blur to the video stream while still temporarily allowing the background replacement visual effect to be temporarily applied until the background replacement visual effect is disabled. Thus, some frames of the video stream may contain both visual effects, but this approach ensures that unprocessed frames are not forwarded to other client devices participating in the video conferencing session.

[0046] According to example embodiments disclosed herein, a client device may be configured to differentiate between a "green room" or "waiting room" state and a "joined" state when switching between a first mode and a second mode, wherein the user is waiting to enter or join a meeting in a video conference, and wherein the user has entered or joined the meeting. For example, in response to the client device moving from the waiting room to the joined state, one or more processors of the client device may be configured to switch from the second mode to the first mode such that the video processing server is responsible for modifying the video stream generated by the client device using visual effects. For example, a background blur effect (or other visual effect) applied by the client device to the video stream while the client device is waiting in the waiting room to enter a meeting "moves" from the client to the video processing server after the client device enters the meeting. Thus, the video processing server is responsible for applying the background blur effect (or other visual effect) when the client device joins the meeting.

[0047] According to example embodiments disclosed herein, a video conferencing application may have a default setting for a first mode. That is, the video conferencing application may prefer to apply visual effects by the video processing server in the first mode (rather than by the client device in the second mode) during a video conferencing session by default. In response to the video processing server being unavailable (e.g., due to resource interruption, peak usage, bandwidth limitations, etc.), one or more processors of the client device may be configured to switch to a second mode, through which the client device applies processing of visual effects. For example, in response to the video processing server running out of graphics computing resources, the intermediate server is configured to signal back to the client device (e.g., via a message or via metadata): the first mode (i.e., in which the visual effects are applied by the video processing server rather than the client device) has become unavailable, and the client device will fall back to the second mode in which the client device applies visual effects (i.e., client-side effects). In an example embodiment, if the client device cannot support the visual effects requested to be applied to the video stream, the one or more processors of the client device may be configured to control the display of the client device to display a message indicating that the requested visual effects cannot be applied to the video stream, and the camera of the client device will be turned off to protect user privacy. The client device may notify the user by additional or alternative methods (eg, through a speaker) that the requested visual effect cannot be applied to the video stream.

[0048] In an example embodiment, if a client device requests a video processing server to apply a visual effect to a video stream and the video processing server is unable to modify the video stream (e.g., due to a resource outage or processing limitation), the video processing server may be configured to forward the video stream to other client devices in the video conferencing session without applying the visual effect. However, in an example embodiment, if the video processing server is unable to modify the video stream (e.g., due to a resource outage or processing limitation), the video processing server may also be configured not to forward the video stream to other client devices in the video conferencing session. The video processing server may receive an instruction from the client device: if the video processing server is unable to modify the video stream as requested, do not forward the video stream to other client devices in the video conferencing session. This aspect provides additional privacy for the user of the client device in the case where the requested visual effect is privacy-related (e.g., the user may not want other users in the video conferencing session to see a non-blurred background). Therefore, during intermittent failures, remote client devices in the video conferencing session may receive a modified video stream, or may not receive video at all.

[0049] According to the example embodiments disclosed herein, a client device may be configured to display a self-view of a user of the client device during a video conferencing session. For example, when the client device operates in a first mode and requests a video processing server to apply visual effects to a video stream, the client device may be configured to remotely receive a modified video stream. Due to network round trips and transcoding performed on the video processing server, the modified video stream received remotely may have worse quality and latency than the local self-view generated by the client device. However, according to the example embodiments disclosed herein, the quality and latency of the self-view may be improved according to various techniques. Therefore, the media quality and latency of the self-view may be maintained at a level that does not negatively affect the user's perception of the quality of the video conferencing session. For example, in order to obtain a lower latency (e.g., a threshold of less than one second, a threshold of less than 300 mms, etc.), when sending the modified video stream back to the client device, the video processing server may be configured to enable WebRTC low-latency rendering and set a threshold limit for latency (e.g., by setting a threshold limit in the PlayoutDelayLimits RTP header extension). As an additional or alternative means of improving the quality of the self-view, the video processing server can be configured to adjust the bit rate allocation policy so that sending the modified video stream back to the client device takes precedence over sending the modified video stream to other client devices. That is, the video processing server can be configured to prioritize the looped back modified video stream when distributing available bandwidth downstream. Without such prioritization, a user of a client device may lose his or her self-view when not speaking in a larger meeting, but the user may still be visible to others in the video conferencing session. As an additional or alternative means of improving the quality of the self-view, one or more processors of the client device can be configured to reduce the frame rate of a remotely received video stream (e.g., a video stream transmitted from other client devices in the video conferencing session) as performance adaptation, wherein the self-view modified video stream is not included (i.e., maintained at a default frame rate). Despite implementing one or more of the above techniques (e.g., by increasing the priority of a modified video stream to be sent back to the client device), if there is still insufficient bandwidth for the looped modified video stream, the one or more processors of the client device may be configured to detect that the video processing server and / or the intermediate server is unable to modify the video stream generated by the client device using the visual effect, and the client device may switch from processing the video in a first mode to processing the video in a second mode in which the client device modifies the video stream by applying the visual effect to the video stream.

[0050] According to example embodiments disclosed herein, a client device may be configured to toggle or switch between a first mode and a second mode based on the size of a self-view and / or the physical resolution of a display of the client device, in which the client device requests a video processing server to apply visual effects to a video stream generated at the client device, and in which the client device applies visual effects to the video stream and transmits the modified video stream to the video processing server. The size of the self-view may correspond to the active resolution of the self-view. For example, during a video conferencing session, when the size of the self-view displayed on the display of the client device is greater than a threshold, one or more processors of the client device are configured to switch to (or maintain) a second mode in which the client device applies visual effects to the video stream and transmits the modified video stream to the video processing server. When the size of the self-view is greater than a threshold, a user-perceivable video quality problem may occur at the client device. Conversely, when the size of the self-view displayed on the display of the client device is less than a threshold, one or more processors of the client device are configured to switch to (or maintain) a first mode in which the client device requests a video processing server to apply visual effects to a video stream generated at the client device.

[0051] In an embodiment, during a video conferencing session, when multiple self-views are displayed on a display of a client device, one or more processors of the client device are configured to determine a self-view with a maximum size. The one or more processors of the client device are configured to determine whether the size of the self-view with the maximum size is greater than a threshold value to determine whether to switch to (or maintain) a second mode in which the client device applies a visual effect to a video stream and transmits the modified video stream to a video processing server. The size of the self-view may correspond to the active resolution of the self-view with the maximum size. One or more processors may be configured to track or monitor the size of one or more self-views during the video conferencing session to determine whether the maximum size changes and whether to switch to (or maintain) a second mode in which the client device applies a visual effect to a video stream and transmits the modified video stream to a video processing server. Depending on whether the client device is switched to the first mode or the second mode, changes in the size and / or position of the self-view may be transmitted to the video processing server to update the position of the visual effect applied to the self-view. In an example, when the user frequently changes the size of the self-view, one or more processors may be configured to wait until resizing stops for a predetermined amount of time (e.g., three seconds) and then determine whether the size of the self-view displayed on the display of the client device is greater than a threshold.

[0052] Intermittent video issues (video freezing, dropped frames, etc.) may occur when a client toggles or switches between a first mode (in which the client device requests a video processing server to apply visual effects to a video stream generated at the client device) and a second mode (in which the client device applies the visual effects to the video stream and transmits the modified video stream to the video processing server). Such intermittent video issues may be more noticeable in the self-view than in the remote participants of the video conferencing session. In an example embodiment, when switching from one implementation to another (i.e., from the first mode to the second mode or vice versa), the client device is configured to wait until the new implementation has begun (i.e., the client device begins receiving remote frames of the self-view generated by the video processing server, or the visual effects are applied by the client device) and then shut down the previous implementation.

[0053] For example, when switching from the second mode to the first mode, one or more processors of the client device can be configured to keep displaying the self-view through the local stream until the first key frame of the remote modified video stream is received from the video processing server. This approach can result in no freezing and minimal frames of double application of effects (e.g., double blurring due to both the client device and the video processing server applying visual effects). If the video stream has not been resized for multiple resolutions, the remote view at other client devices in the video conferencing session may freeze when the visual effects are requested from the video processing server, and once the first key frame is propagated from the client device through the video processing server, the remote view will unfreeze. Since the client device will not stop applying visual effects until it has seen the new modified video stream, there will be some frames of double application of effects. As the video stream restarts, both the client device and the remote client device participating in the video conferencing session can see a decrease in quality.

[0054] For example, when switching from the first mode to the second mode, one or more processors of the client device can be configured to immediately display the self-view using the local stream once the visual effect is applied by the client, thereby hiding any artifacts. The remote view at other client devices in the video conferencing session can show some frames with the effect double applied (e.g., double blurred) until the visual effect is disabled by the video processing server, and some pauses / dropped frames can be shown while the client device loads the visual effect.

[0055] In some implementations, in order to reduce freeze time and / or reduce or avoid frames with double effects, other configurations at the client device and / or video processing server can be implemented. For example, the video processing server can remain in a state (e.g., ready state) ready to apply visual effects to the video stream transmitted from the client device but not actually apply any visual effects until the mode is switched to the first mode. Keeping the video processing server in a state ready to apply visual effects reduces the startup time of the video processing server and reduces the freeze time of the remote client device in the video conferencing session. In an embodiment, the client device can be configured to send a timestamp / frame counter to the video processing server, indicating the future time when the client device wants to switch modes. The video processing server is configured to ensure that the visual effects are enabled / disabled from the frame at the appropriate time. The client device is configured to wait until the timestamp / frame counter value, and then start / stop applying visual effects, thereby preventing double effect frames. According to this embodiment, in this embodiment, the client device sends a timestamp / frame counter, and the video processing server can be configured to switch modes immediately and / or can remain in a ready state.

[0056] In an embodiment, the client device may be configured to send metadata about frames of the video stream that provides information about the visual effect to the video processing server. When the client device requests the video processing server to apply the visual effect, the video processing server is configured to generate a modified video stream with the visual effect only if the applied visual effect is different from any visual effect applied to the incoming video stream. According to the embodiment in which the client device sends metadata about frames of the video stream that provides information about the visual effect to the video processing server, the video processing server may be configured to switch modes immediately and / or may remain in a ready state.

[0057] For example, in some implementations, the client device and / or the video processing server can preload visual effects without actually applying the visual effects to the output stream. When switching from the first mode to the second mode, for example, the client device can be configured to preload visual effects, request the video processing server to freeze at the current frame with the visual effects applied by the video processing server, send a video stream with the visual effects applied by the client device, and then request the video processing server to stop applying the visual effects to the video stream. Therefore, a short video freeze is applied instead of a frame with a double effect. When switching from the second mode to the first mode, for example, the video processing server can be configured to preload visual effects and receive a request from the client device to freeze the frame with the visual effects applied by the client device. The client device can stop applying visual effects to the video stream and send a video stream without visual effects to the video processing server. Then request the video processing server to apply the visual effects to the video stream. Therefore, a short video freeze is applied instead of a frame with a double effect.

[0058] In an example embodiment, a hybrid operation mode can be implemented between a client device and a video processing server. For example, when switching to enable the client device to apply visual effects to a video stream, one or more processors of the client device can be configured to display a self-view using the video stream with visual effects applied to the video stream by the client device, while sending the unmodified video stream to the video processing server, so that the video processing server continues to apply visual effects to the unmodified video stream, and sends the modified video stream (with visual effects applied by the video processing server) to be displayed at a remote client device participating in the video conference session. This means completely hides the switching of the remote client device from the first mode, and for the self-view at the client device, there may be a temporary glitch when switching to the hybrid mode due to the different timelines of the video streams. In an implementation, in order to avoid running in a hybrid mode for a long period of time, the client device can be configured to switch from the hybrid mode to the second mode after a predetermined period of time. Once the size of the self-view is stabilized below a threshold, this means prevents the waste of cloud resources, so that the client device can apply visual effects without degrading the self-view too much.

[0059] In another example embodiment, viewing requests for the self view may be limited to a threshold resolution (eg, 640x360) to make switching from the second mode to the first mode on the self view faster.

[0060] In another example embodiment, when any zoom feature (e.g., auto zoom or center feature) is enabled in the video conferencing application at the client device, switching to the first mode may be disabled. However, in an embodiment, the video processing server may be configured to apply visual effects to the video stream in the first mode when the zoom feature is enabled in the video conferencing application at the client device. For example, a scaled stream (i.e., an original video stream modified by zooming at the client) may be sent from the client device to the video processing server, and the visual effects may be applied to the scaled stream. For example, an original video stream may be sent from a client device to a video processing server, and the video processing server may apply both zooming and visual effects to the video stream.

[0061] The present disclosure provides multiple technical effects and benefits. Video processing is an expensive processing task (e.g., a CPU / GPU heavy task) that consumes a large amount of bandwidth of client devices such as laptops, tablet computers, smart phones, etc. When executing video conferencing applications, adding visual effects to video streams can cause client devices to overheat, consume batteries, and / or client devices to have difficulty in achieving high-quality visual effects (e.g., frames per second drop or client devices slow down). As an example technical effect and benefit, the system and method disclosed herein allow certain video processing to be offloaded to the cloud (e.g., to a server, such as a video processing server), which releases pressure on client devices. Therefore, providing a server such as a video processing server enables video conferencing to run smoothly at client devices with high quality and high efficiency, and the server is configured with hardware to modify the video stream transmitted by the client device to include visual effects requested by the client device. Technical benefits include reduced client CPU usage, increased stability, increased bandwidth, better video quality (e.g., higher resolution and / or frame rate), and improved user experience.

[0062] Now referring to the accompanying drawings, Figure 1 is an example video conferencing system according to one or more example embodiments of the present disclosure. Figure 1An example of a video conferencing system is shown, which includes a first client device 100, a second client device 100', a third client device 100'', a first server (first intermediate server) 200, a second server (second intermediate server) 200', a third server (third intermediate server) 200'' and a video processing server 300, each of which can communicate with each other through a network. For example, the first client device 100, the second client device 100' and the third client device 100'' may include any one of a personal computer, a smart phone, a laptop computer, a tablet computer, etc. The network may include any type of communication network such as a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), etc. For example, wireless communication between the elements of the example embodiment may be performed through wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), Bluetooth low energy (BLE), near field communication (NFC), radio frequency (RF) signals, etc. For example, wired communications between elements of example embodiments may be performed via twisted pair cables, coaxial cables, fiber optic cables, Ethernet cables, or the like.

[0063] As will be explained in more detail below, in some implementations, the first client device 100 and one or more other client devices (e.g., the second client device 100', the third client device 100", etc.) can participate in a video phone (hereinafter referred to as a video conference) session using the video processing server 300 over a network. The video conference session may include multiple client devices, including Figure 1 Client devices other than the first client device 100 , the second client device 100 ′, and the third client device 100 ″ are shown.

[0064] exist Figure 1 In the example, the first client device 100 transmits a video stream 190 (e.g., a raw video stream) to the video processing server 300 through the first server 200. The video stream 190 may correspond to a self-view of a user of the first client device 100 who is participating in or waiting to participate in a video conferencing session with other client devices. In an example, the video stream 190 may be a raw video stream in which visual effects have not yet been applied to the video stream (e.g., a blurred background and / or lighting effects are not applied to the user's self-view). In addition to the video stream 190, in an example, the first client device 100 also transmits visual effect information 190' (e.g., media features) corresponding to the visual effects to be applied to the video stream by the video processing server 300 to the video processing server 300 through the first server 200.

[0065] In response to receiving the video stream 190 and the visual effect information 190', the video processing server 300 modifies the video stream 190 by adding one or more visual effects to the video stream 190 based on the visual effect information 190'. The video processing server 300 transmits the modified video stream to other client devices participating in the video conference session. For example, the video processing server 300 transmits the modified video stream 192 to the second client device 100' through the second server 200', and transmits the modified video stream 194 to the third client device 100' through the third server 200''. In some implementations, the video processing server 300 can transmit the modified video stream (loopback stream) 196 back to the first client device 100 through the first server 200. In some implementations, the modified video streams 192, 194, 196 can be the same (e.g., have the same resolution), while in other implementations, the modified video streams 192, 194, 196 can be different from each other (e.g., have different resolutions). Further details regarding the operation of the video processing server 300 are discussed below.

[0066] FIG. 2A to FIG. 2BAn example flow chart of a video processing server according to one or more example embodiments of the present disclosure is depicted. For example, when the first client device 100 applies visual effects to the video stream itself, the video processing server 300 can be bypassed and the video stream can be transmitted to the client devices participating in the video conferencing session using other servers in the computing network (server system). In another example embodiment, when the video processing server 300 is not requested or required to apply visual effects to the incoming video stream, the video processing server 300 is configured to decode the video stream received from the first client device 100 and pass the decoded video stream to a plurality of resizer nodes (resizeers), each resizer node for each unique resolution requested by the corresponding client device to receive the video stream of the video conferencing session. That is, the video processing server 300 is configured to perform video transcoding operations using a node graph. Transcoding refers to operations such as decompressing (e.g., decoding) a video stream, changing the decompressed video stream (e.g., applying one or more visual effects, resizing the video stream, etc.), and recompressing (e.g., encoding) the video stream. In an implementation where the video processing server 300 does not apply visual effects, a client device requesting a video stream with the highest resolution can receive the video stream from the client device through another server without the video stream passing through the video processing server 300 because no resizing operation needs to be performed by the video processing server 300. According to another example, the video processing server is configured to pass the decoded video stream to a plurality of resizer nodes, wherein each of the resizer nodes is bucketed to correspond to a resolution at a specific bit rate for a corresponding client device participating in the video conferencing session, which may or may not correspond exactly to the resolution of the corresponding client device. For example, the corresponding client device may have the ability to process a high-definition (HD) video stream (e.g., 1920x1080), but may be limited to a specific resolution (e.g., 800x600, 640x360, etc.) due to bit rate and / or bandwidth limitations, and therefore the video processing server 300 can resize the video stream according to the specific resolution based on the bit rate of the corresponding client device.

[0067] refer to Figure 2A , the video processing server 300 may be configured to process an incoming video stream 210 received from the first client device 100, wherein visual effects have been applied to the incoming video stream 210 on the client side (ie, by the first client device 100). Figure 2AIn the example of , the video processing server 300 processes the incoming video stream 210 by decoding the incoming video stream 210 through the decoder 332. The video processing server 300 is also configured to resize the decoded video stream by performing a resizing operation through the resizers 334a, 334b ​​to obtain a desired resolution of the incoming video stream 210. As described above, the desired resolution can be based on the capabilities of the client device that is to receive the video stream from the video processing server 300, or based on existing bit rate and / or bandwidth limitations of the client device that is to receive the video stream from the video processing server 300. For example, the resizer 334a can be configured to resize the decoded video stream using a first resolution (e.g., 640x480), and the resizer 334b ​​can be configured to resize the decoded video stream using a second resolution (e.g., 320x240). The encoder 336a is configured to encode the video stream resized by the resizer 334a to output a first output video stream 220a, and the encoder 336b is configured to encode the video stream resized by the resizer 334b ​​to output a second output video stream 220b. For example, the first output video stream 220a can be transmitted to the second client device 100' through the second server 200', and the second output video stream 220b can be transmitted to the third client device 100'' through the third server 200''.

[0068] refer to Figure 2B , the video processing server 300 is shown to include a visual effects applicator 338. The visual effects applicator 338 also includes Figure 2A , but are omitted for clarity. Figure 2A In the embodiment of , the visual effect applier 338 may be bypassed or do not perform any operations on the video stream because the visual effect has been applied by the first client device 100 and the video stream is only passed to the resizers 334a, 334b. Figure 2B In an embodiment of the present invention, the video processing server 300 can be configured to process an incoming video stream 230 received from a first client device 100, wherein no visual effects have been applied to the incoming video stream 230 on the client side (i.e., by the first client device 100), but rather a visual effect applier 338 applies one or more visual effects to the incoming video stream 230 (i.e., on the cloud side). Figure 2BIn the example of , the video processing server 300 processes the incoming video stream 230 by decoding the incoming video stream 230 via the decoder 332. The visual effect applier 338 is configured to apply one or more visual effects to the decoded video stream according to the visual effect information 230' also received from the first client device 100. The visual effect information 230' may include information related to one or more settings of the visual effect to be applied to the incoming video stream 230. For example, the one or more settings may include a blur radius of a background blur effect to be applied to at least a portion of the incoming video stream 230. For example, the one or more settings may include a brightness level of a lighting effect to be applied to at least a portion of the incoming video stream 230. Figure 2A Similarly, the video processing server 300 is configured to resize the decoded video stream (with the visual effects applied by the visual effects applier 338) via one or more resizers. Figure 2BIn the example of , three resizers 334a, 334b, 334c are shown. The resizers 334a, 334b, 334c are configured to perform resizing operations to obtain a desired resolution of the incoming video stream 230. The desired resolution may be based on the capabilities of the client device that is to receive the video stream from the video processing server 300, or based on existing bit rate and / or bandwidth limitations on the client device that is to receive the video stream from the video processing server 300. For example, the resizer 334a may be configured to resize the decoded video stream using a first resolution (e.g., 640x480), the resizer 334b ​​may be configured to resize the decoded video stream using a second resolution (e.g., 320x240), and the resizer 334c may be configured to resize the decoded video stream using an HD resolution (e.g., 1920x1080). The encoder 336a is configured to encode the video stream resized by the resizer 334a to output a first output video stream 240a, the encoder 336b is configured to encode the video stream resized by the resizer 334b ​​to output a second output video stream 240b, and the encoder 336c is configured to encode the video stream resized by the resizer 334c to output a third output video stream 240c. For example, the first output video stream 240a may be transmitted to the second client device 100' via the second server 200', the second output video stream 240b may be transmitted to the third client device 100' via the third server 200'', and the third output video stream 240c may be transmitted to the first client device 100 via the first server 200. However, the present disclosure is not limited to these example resolution values, and in some implementations, the resolution of the video stream received from the first client device 100 may not be modified or resized. The present disclosure is also not limited to the example client device described as receiving the output video stream, and the output video stream may be transmitted to client devices participating in the video conferencing session other than the example client device described.

[0069] Reference now Figure 3 , an example block diagram of a client device and a video processing server according to one or more example embodiments of the present disclosure will now be described. Figure 3 denoted as the first client device 100 , but the features of the first client device 100 described herein are also applicable to the second client device 100 ′ and the third client device 100 ″.

[0070] First client device 100 may include one or more processors 110, one or more memory devices 120, video conferencing application 130, camera 140, input device 150, and display 160. Video processing server 300 may include one or more processors 310, one or more memory devices 320, and video conferencing service provider 330.

[0071] For example, the one or more processors 110, 310 may be any suitable processing device that may be included in the first client device 100 or the video processing server 300. For example, the processor 110, 310 may include one or more of a processor, a processor core, a controller and an arithmetic logic unit, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an image processor, a microcomputer, a field programmable array, a programmable logic unit, an application specific integrated circuit (ASIC), a microprocessor, a microcontroller, etc. and combinations thereof, including any other device that can respond and execute instructions in a defined manner. The one or more processors 110, 310 may be a single processor or a plurality of processors that are operably connected (e.g., in parallel).

[0072] The memory 120, 320 may include one or more non-transitory computer-readable storage media, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM) and flash memory, USB drive; volatile memory devices, such as random access memory (RAM), hard disk, floppy disk, Blu-ray disk or optical media such as CDROM disk and DVD; and combinations thereof. However, examples of the memory 120, 320 are not limited to the above description, and the memory 120, 320 may be implemented by other various devices and structures as will be understood by those skilled in the art.

[0073] For example, the memory 120 may store instructions that, when executed, cause the one or more processors 110 to execute the video conferencing application 130 with another client device through the video processing server 300, as described in an example according to the present disclosure. For example, the memory 320 may store instructions that, when executed, cause the one or more processors 310 to provide a video conferencing service for multiple client devices, as described in an example according to the present disclosure.

[0074] The memory 120 may also include data 122 and instructions 124 that may be retrieved, manipulated, created, or stored by the one or more processors 110. In some example embodiments, this data may be accessed and used as input to perform a video conferencing application 130 with another client device through the video processing server 300, as described in examples according to the present disclosure. The memory 320 may also include data 322 and instructions 324 that may be retrieved, manipulated, created, or stored by the one or more processors 310. In some example embodiments, this data may be accessed and used as input to provide video conferencing services to multiple client devices, as described in examples according to the present disclosure.

[0075] The first client device 100 includes a video conferencing application 130, which may also be referred to as a video phone application or a video calling application. The video conferencing application 130 enables a user of the first client device 100 to communicate with a user of another client device through the transmission of audio and visual signals, thereby allowing two or more users to participate in a real-time conversation from different locations, wherein the two or more users can view each other through a video stream. According to an example of the present disclosure, the video conferencing application 130 may include a self-view size determiner 132 and a visual effect mode selector 134. The features of the self-view size determiner 132 and the visual effect mode selector 134 will be described in more detail below.

[0076] The first client device 100 includes a camera 140, from which the video conferencing application 130 can capture image data and generate a video stream, which is transmitted to the video processing server 300. In some example embodiments, the camera 140 collects image data from one or more users of the first client device 100, such as a self-view of one or more users. The camera 140 can be any device capable of capturing visual data. The first client device 100 is configured to generate a video stream and send the video stream to the video processing server 300 (e.g., via one or more intermediate servers, such as the first server 200, via the network 400). For example, the camera 140 can be an integrated network camera of the first client device 100, or can be a camera device connected in communication, etc. The first client device 100 can encode the captured video (e.g., as specified by the instructions 124, etc.). In some implementations, the first client device 100 can encode the captured video at a high resolution and high quality (e.g., in HD format).

[0077] The first client device 100 includes an input device 150, which is configured to receive input from a user and may include, for example, one or more of the following: a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus, a gesture recognition sensor (e.g., for recognizing a user's gesture, including the movement of a body part), an input sound device or a voice recognition sensor (e.g., a microphone for receiving voice commands), an output sound device (e.g., a speaker), a trackball, a remote controller, a portable (e.g., a cellular or smart) phone, a tablet PC, a pedal or foot switch, a virtual reality device, etc. The input device 150 may further include a tactile device for providing tactile feedback to the user. For example, the input device 150 may also be embodied by a touch-sensitive display with touch screen capabilities. The input device 150 may be used by a user of the first client device 100 to select one or more visual effects to be applied to the video stream generated by the first client device 100. For example, input device 150 may be used by a user of first client device 100 to select one or more visual effects to be applied to the user's self-view, such as lighting effects, background blur effects, background replacement effects, etc. Input device 150 may be used by a user of first client device 100 to select a default setting regarding when to apply visual effects on the client side or on the cloud side. For example, a user may specify a default setting that, upon initial execution of video conferencing application 130, video processing server 300 is to apply visual effects for a video conferencing session.

[0078] The first client device 100 includes a display 160 that displays information viewable by a user. For example, the display 160 may be a non-touch-sensitive display or a touch-sensitive display. For example, the display device 160 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), a cathode ray tube (CRT) display, etc. However, the present disclosure is not limited to these example displays, and may include other types of displays.

[0079] According to example embodiments described herein, the video processing server 300 may include one or more processors 310 and a memory 320 previously discussed above. The video processing server 300 may also include a video conferencing service provider 330. For example, the video conferencing service provider 330 may include a transcoder 331 and a visual effects applier 338. The transcoder 331 may be configured to receive compressed media (e.g., video) from the first client device 100, decompress the media, and apply visual effects to the uncompressed media. In order to implement visual effects to the video stream, the transcoder 331 may utilize a library that supports combining visual rendering effects (e.g., blur, zoom, lighting, etc.) with the execution of foreground segmentation (e.g., in the self-view of the user of the first client device 100, segmenting out the user and applying visual effects to the background). For example, the transcoder 331 may include one or more decoders 332 to decode the video stream received from the first client device 100, such as with respect to FIG. 2A to FIG. 2B For example, the transcoder 331 may include one or more resizers 334 to resize the video stream received from the first client device 100, such as FIG. 2A to FIG. 2B For example, the transcoder 331 may include one or more encoders 336 to encode each respective video stream that has been resized by the corresponding resizer 334 and then output it to a respective client device participating in the video conferencing session, for example, as described with respect to FIG. 2A to FIG. 2B The embodiments discussed above.

[0080] The visual effect applier 338 is configured to apply one or more visual effects to a video stream received from the first client device 100 based on the visual effect information received from the first client device 100. For example, the visual effect information may be received in the form of configuration properties associated with the video stream (e.g., via metadata). The configuration properties may include one or more settings for one or more visual effects to be applied to the video stream by the video processing server 300. When the first client device 100 requests the video processing server 300 to apply a visual effect to the video stream, the decoder 332 decodes the video stream received from the first client device 100, passes the decoded video stream to the visual effect applier 338 (also referred to as a media pipeline effects module) to apply the visual effect to the video stream to generate a modified video stream.

[0081] The video processing server 300 may be a specially configured server or server group dedicated to performing video processing on a video stream received from a client device participating in a video conferencing session in order to apply visual effects to the video stream. For example, the video processing server 300 may include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or applying visual effects to the video stream), and multiple video encoder accelerators (e.g., ASICs for performing transcoding operations such as encoding, decoding, and resizing).

[0082] will be in view of Figures 4 to 5 The following diagram and Figures 6 to 10 Additional aspects of the first client device 100 and the video processing server 300 are discussed with reference to the flowchart of FIG.

[0083] Figure 4 Depicted is an example display of a video conferencing session at a client device according to one or more example embodiments of the present disclosure. Figure 4 During the video conference session 410, when the video conference application 130 is executed at the first client device 100, the display 160 may display a presentation 420, a self-view 430 of a user of the first client device 100, and one or more external live streams 440 of one or more other users of other client devices participating in the video conference session 410. For example, the first client device 100 may receive one or more video streams from the video processing server 300, the first server 200, or another server not specifically shown in the figure. The position and / or size of each of the presentation 420, the self-view 430, and / or the external live stream 440 may be changed, for example, according to input received from the user of the first client device 100. For example, the user of the first client device 100 may change the size of the self-view 430 through the input device 150. In addition, one or more of the presentation 420, the self-view 430, and the one or more external live streams 440 may not be displayed at the display 160 of the first client device 100. In addition, although Figure 4 The user of first client device 100 is indicated as being the presenter of presentation 420, but the user of first client device 100 may be any participant in video conference session 410, not necessarily the presenter.

[0084] Figure 5 Depicted is an example display of a visual effect applied to a user's self-view in a video conferencing session at a client device according to one or more example embodiments of the present disclosure. Figure 5, during a video conference session, when the video conference application 130 is executed at the first client device 100, the display 160 may display a self-view 510 of the user of the first client device 100, wherein the self-view 510 includes the user 520 and the blurred background 530. Figure 5 In the example self-view 510 of FIG. 5 , a visual effect has been applied to the background of the user 520, causing the background to be blurred, which the user 520 may select or request, for example, for privacy reasons. The visual effect may be applied to the background by the first client device 100 or the video processing server 300 according to the operating mode of the first client device 100, according to the size of the user's self-view, and various other factors discussed below. The present disclosure is not limited to Figure 5 For example, visual effects other than blur effects can be applied to a video stream, or visual effects can be applied to a video stream in addition to blur effects. For example, lighting effects (e.g., increasing or decreasing brightness and / or contrast values, etc.), replacing backgrounds, and other visual effects can be applied to a video stream.

[0085] Figures 6 to 10 A flowchart of an example non-limiting computer-implemented method is shown according to one or more example embodiments of the present disclosure.

[0086] refer to Figure 6 , method 600 includes operation 610 in which a server system (e.g., video processing server 300) receives a video stream related to a video conferencing session from a first client device 100. The method further includes operation 620 in which the server system (e.g., video processing server 300) receives visual effect information related to one or more visual effects to be applied to the video stream from the first client device 100. Operation 630 includes the server system (e.g., video processing server 300) applying the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams (e.g., modified video streams 192, 194). Operation 640 includes the server system (e.g., video processing server 300) transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session (e.g., second client device 100' and / or third client device 100''). Although Figure 6 Not shown, but optional operations may include a server system (e.g., video processing server 300) applying one or more visual effects to a video stream based on the received visual effects information to generate a modified video stream (e.g., looped or modified video stream 196), and transmitting the modified video stream to the first client device 100.

[0087] For the modified video stream 196 (i.e., the looped video stream), the first client device 100 may be configured to receive the modified video stream 196 from the video processing server 300 by specifically requesting the video processing server 300 to transmit the modified video stream 196 back to the first client device 100, or the video processing server 300 may be configured to automatically transmit the modified video stream 196 back to the first client device 100. For example, when the visual effect is requested by the first client device 100, the video processing server 300 is configured to signal (e.g., through a message or through metadata) to the intermediate server (e.g., the first server 200) that the video stream received from the first client device 100 will also be at least partially used for transmission from the video processing server 300 to the first client device 100 through the intermediate server. Therefore, in response to receiving the signal from the video processing server 300, the intermediate server prepares to send at least a portion of the video stream back to the first client device 100, and sends a push notification to the first client device 100 to notify the first client device 100 about the new stream. When the first client device 100 receives notification of the modified stream, the first client device 100 is configured to perform a search of incoming data from the intermediate server that includes the modified video stream 196. The first client device 100 may be configured to prioritize displaying the modified video stream 196 over the video stream 190 generated locally at the first client device 100.

[0088] In an example embodiment, if the first client device 100 requests the video processing server 300 to apply one or more visual effects to the video stream 190 and the video processing server 300 is unable to modify the video stream 190 (e.g., due to resource interruptions or processing limitations), the video processing server 300 may be configured to forward the video stream 190 to other client devices in the video conference session (e.g., the second client device 100' and / or the third client device 100'') without applying the visual effects. However, in an example embodiment, if the video processing server 300 is unable to modify the video stream 190 (e.g., due to resource interruptions or processing limitations), the video processing server 300 may also be configured not to forward the video stream 190 to other client devices in the video conference session (e.g., the second client device 100' and / or the third client device 100''). The video processing server 300 may receive an instruction from the first client device 100 not to forward the video stream 190 to other client devices (e.g., the second client device 100' and / or the third client device 100'') in the video conferencing session if the video processing server 300 is unable to modify the video stream 190 as requested. This aspect provides additional privacy for the user of the first client device 100 in situations where the requested one or more visual effects are privacy-related (e.g., the user may not want other users in the video conferencing session to see a non-blurred background). Thus, other (remote) client devices in the video conferencing session may receive a modified video stream (with one or more visual effects applied by the video processing server 300 as requested by the first client device 100), or may not receive any video at all from the first client device 100 during an intermittent failure of the video processing server 300.

[0089] refer to Figure 7 , method 700 includes an operation 710 in which a computing device (e.g., first client device 100) determines whether to process a video related to a video conferencing session in a first mode or a second mode. For example, first client device 100 may be configured to toggle or switch between a first mode in which first client device 100 requests video processing server 300 to apply a visual effect to a video stream 190 generated at first client device 100 and a second mode in which first client device 100 applies the visual effect to the video stream itself and processes the modified video stream (e.g., Figure 2AThe first client device 100 may transmit the modified video stream 210 (e.g., via the first server 200) to the video processing server 300, which then resizes the modified video stream as needed and outputs the resized modified video stream to one or more other client devices participating in the video conferencing session. In some cases, the first client device 100 may transmit the modified video stream to the one or more other client devices participating in the video conferencing session without the modified video stream passing through the video processing server (e.g., via one or more servers, such as any of the first server 200, the second server 200', the third server 200''). In some cases, a remote client device requesting a video stream with the highest resolution may receive the modified video stream from the first client device 100 via another server (e.g., any of the first server 200, the second server 200', the third server 200'') without the modified video stream passing through the video processing server 300 (because the resizing operation does not need to be performed by the video processing server 300).

[0090] In response to determining to process the video in the first mode, at operation 720, the method includes the computing device (e.g., the first client device 100) generating a first video stream and transmitting the first video stream (e.g., the incoming video stream 230) to the server (e.g., the video processing server 300), and at operation 730, the method includes the computing device (e.g., the first client device 100) transmitting visual effect information (e.g., visual effect information 230') related to one or more visual effects to be applied to the first video stream (e.g., the incoming video stream 230) by the video processing server 300 to the server (e.g., the video processing server 300). Here, the first video stream (e.g., the incoming video stream 230) corresponds to an original video stream generated by image data captured by the camera 140, which may correspond to a self-view of the user. Therefore, in the first mode, the first client device 100 transmits the original video stream without visual effects to the video processing server 300, and additionally transmits visual effect information related to one or more visual effects to be applied to the first video stream to the video processing server 300.

[0091] In response to determining to process the video in the second mode, at operation 780, the method includes applying one or more visual effects to the first video stream by the computing device (e.g., the first client device 100) (before the first video stream is actually transmitted), and the computing device (e.g., the first client device 100) generates a second video stream (e.g., the incoming video stream 210). At operation 790, the method includes the computing device (e.g., the first client device 100) transmitting the second video stream (e.g., the incoming video stream 230) to a server (e.g., the video processing server 300) or transmitting the second video stream to another server (e.g., a separate server system, which may include any of the first server 200, the second server 200', the third server 200', etc.), thereby completely bypassing the video processing server 300.

[0092] In some implementations, a computing device (e.g., a first client device 100) may be configured to operate in a hybrid mode. For example, when the first client device 100 is to switch from a first mode to a second mode in which the first client device 100 applies visual effects to a video stream, the first client device 100 may operate in a hybrid mode. If it is determined at operation 740 that the first client device 100 is to operate in a hybrid mode (e.g., before switching to the second mode), at operation 760, one or more processors 110 of the first client device 100 are configured to generate a second video stream by applying one or more visual effects to the first video stream by the first client device 100, and at operation 770, the first client device 100 displays the second video stream as a self-view. At the same time, the first client device 100 transmits the first video stream and the visual effect information to a server (e.g., a video processing server 300). That is, when the first client device 100 is in hybrid mode, the unmodified video stream and visual effect information continue to be sent to the video processing server 300, so that the video processing server 300 continues to apply one or more visual effects to the unmodified video stream, and one or more modified video streams (with visual effects applied by the video processing server 300) are sent to be displayed at the remote client devices (e.g., the second client device 100' and / or the third client device 100") participating in the video conference session. Implementing the hybrid mode completely hides the switching of the remote client device from the first mode to the second mode, and for the self-view at the first client device 100, there may be minimal interruption or glitches when switching to the hybrid mode due to the different timelines of the video streams. In an implementation, in order to avoid operating in the hybrid mode for a long period of time, the first client device 100 can be configured to switch from the hybrid mode to the second mode after a predetermined period of time (e.g., after five minutes). Once the size of the self-view is stabilized below the threshold, this means prevents the waste of cloud resources, so that the first client device 100 can apply visual effects without degrading the self-view too much.

[0093] If it is determined at operation 740 that the first client device 100 does not operate in the hybrid mode, then at operation 750, the first client device 100 receives a modified video stream (e.g., modified video stream 196) from a server (e.g., video processing server 300) and displays the modified video stream as a self-view on the display 160. For example, the modified video stream includes one or more visual effects applied to the first video stream by the server (e.g., video processing server 300).

[0094] refer to Figure 8, the method 800 includes an operation 810 of a computing device (e.g., a first client device 100) determining whether the size of the self-view is greater than a threshold level. For example, the visual effect mode selector 134 may be configured to toggle or switch between a first mode and a second mode based on the size of the self-view and / or the physical resolution of the display of the first client device 100, in which the first client device 100 requests the video processing server 300 to apply the visual effect to the video stream generated at the first client device 100, and in which the first client device 100 applies the visual effect to the video stream and transmits the modified video stream to the video processing server 300. The size of the self-view may correspond to the active resolution of the self-view and may be determined by the self-view size determiner 132. For example, during a video conferencing session, when the size of the self-view displayed on the display of the client device is greater than a threshold level (value), the operation 820 of the computing device (e.g., the first client device 100) includes the visual effect mode selector 134 determining to switch to (or maintain) the second mode. When the size of the self-view is greater than the threshold level (value), a user-perceivable video quality problem may occur at the client device. In contrast, when the size of the self-view displayed on the display of the first client device 100 is less than a threshold level (value), operation 830 of the computing device (e.g., the first client device 100) includes the visual effect mode selector 134 determining to switch to (or maintain) the first mode.

[0095] In some implementations, in response to the media quality of the modified video stream (e.g., the self-view) being less than a threshold level and / or the latency of the modified video stream received from the video processing server 300 being greater than a threshold level, the one or more processors 110 are configured to determine to switch to the second mode and / or stop transmitting the video stream to the video processing server 300. For example, it may be determined whether the quality of the modified video stream (e.g., the self-view) has dropped below a threshold based on various factors and indicators related to quality of service and quality of experience (e.g., whether the resolution of the self-view has dropped below a threshold level, whether the self-view has been frozen a certain number of times within a predetermined time period, whether the self-view has been frozen for longer than a predetermined time period, a signal-to-noise ratio of the self-view, etc.).

[0096] In some implementations, in response to a media quality of a modified video stream transmitted to the first client device 100 being less than a threshold level and / or a latency of the modified video stream transmitted to the first client device 100 being greater than a threshold level, the video processing server 300 is configured to transmit a notification to the first client device 100 indicating that the first client device 100 will take over control to apply one or more visual effects to the video stream (switch to the second mode) and / or that the video processing server 300 will stop transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session (e.g., the second client device 100' and the third client device 100'').

[0097] In an embodiment, during a video conference session, when multiple self-views are displayed on the display of the first client device 100, the self-view size determiner 132 is configured to determine the self-view with the largest size. The one or more processors 110 of the first client device 100 are configured to determine whether the size of the self-view with the largest size is greater than a threshold level (value) to determine whether to switch to (or maintain) the second mode. The self-view size determiner 132 can be configured to track or monitor the size of one or more self-views during the video conference session to determine whether the maximum size changes and whether to switch to (or maintain) the second mode. When the first client device is in the first mode, the change in the size and / or position of the self-view can be transmitted to the video processing server 300 to update the position of the visual effect applied to the self-view. In an example, when the user frequently changes the size of the self-view (i.e., changes more than a predetermined number of times within a predetermined time period, such as more than three times within ten seconds), the self-view size determiner 132 can be configured to wait until the resizing has stopped for a predetermined amount of time (e.g., three seconds) and then determine the size of the self-view. The one or more processors 110 may then determine whether the size of the self-view displayed on the display of the first client device 100 is greater than a threshold level (value).

[0098] refer to Fig. 9, method 900 includes operation 910 of a computing device (e.g., first client device 100) determining whether a zoom feature (e.g., an automatic zoom or center feature) is enabled in a video conferencing application at the first client device 100. That is, when the one or more processors 110 determine that the zoom feature is enabled, operation 920 may include the visual effects mode selector 134 switching to or maintaining the second mode. For example, the video processing server 300 may not be allowed to apply visual effects to the enlarged video stream in the first mode because applying visual effects and resizing the enlarged video stream may result in poor quality (e.g., loss of resolution) of the modified video stream. When the one or more processors 110 determine that the zoom feature is not enabled, operation 930 may include the visual effects mode selector 134 allowing the first client device 100 to switch to the first mode, or maintaining the first mode.

[0099] In an alternative embodiment, the video processing server 300 may be configured to apply visual effects to the video stream in a first mode when a zoom feature is enabled in a video conferencing application at the first client device 100. For example, a zoomed stream (i.e., an original video stream modified by zooming at the first client device 100) may be sent from the first client device 100 to the video processing server 300, and the visual effects may be applied to the zoomed stream. For example, an original video stream may be sent from the first client device 100 to the video processing server 300, and the video processing server 300 may apply both zooming and visual effects to the video stream.

[0100] refer to Fig.10, method 1000 includes an operation 1010 of one or more processors 110 of a computing device (e.g., first client device 100) determining whether a user of first client device 100 is in a waiting room of a video conference session or has joined the video conference session. For example, first client device 100 may be configured to distinguish between a "green room" or "waiting room" state and a "joined" state when switching between a first mode and a second mode, in which the user is waiting to enter or join a meeting in the video conference session, and in which the user has entered or joined the meeting. For example, in some implementations, when first client device 100 is in the waiting state, video processing server 300 may not support sending and receiving media to first client device 100 while first client device 100 remains in the waiting state. For example, in response to one or more processors 110 determining that first client device 100 has moved from the waiting room to the joined state, at operation 1020, visual effect mode selector 134 may be configured to switch from the second mode to the first mode or maintain the first mode. For example, in response to one or more processors 110 determining that the first client device 100 is located in a waiting room, at operation 1030, the visual effects mode selector 134 may be configured to switch from the first mode to the second mode or maintain the second mode. For example, a background blur effect (or other visual effect) applied to a video stream by the first client device 100 while the first client device 100 is in the waiting room waiting to enter a meeting is "moved" from the first client device 100 to the video processing server 300 in response to the first client device 100 entering the meeting in the video conferencing session. Thus, when the first client device 100 joins the meeting, the video processing server 300 is responsible for applying the background blur effect (or other visual effect) to the video stream. In an alternative embodiment, the video processing server 300 may support sending and receiving media to the first client device 100 while the first client device 100 is in a waiting state, and thus may not perform Fig.10 operation.

[0101] When the first client device 100 toggles or switches between the first mode and the second mode, intermittent video problems (e.g., video freezes, dropped frames, etc.) may occur. Such intermittent video problems may be more noticeable in the self-view than in the remote participants of the video conference session. The example embodiments described below provide additional methods for avoiding or mitigating these video problems that occur when switching between the first mode and the second mode. For example, when switching from the first mode to the second mode (or vice versa), the first client device 100 may be configured to wait until the second mode (or the first mode) has begun and then shut down the previous implementation. For example, when switching from the second mode to the first mode, before the first client device 100 stops applying visual effects to the video stream sent to the video processing server 300, the first client device 100 begins to receive remote frames of the self-view generated by the video processing server 300. This may result in certain frames of the video stream received by the first client device 100 and the remote client devices (e.g., the second client device 100' and the third client device 100'') having a dual visual effect applied to the self-view of the user of the first client device 100. For example, when switching from the first mode to the second mode, the first client device 100 begins generating and displaying frames of its own view having the visual effects applied to the video stream by the first client device 100 before the video processing server 300 stops applying the visual effects to the video stream being sent to the video processing server 300 and looped back to the first client device 100. This may also result in certain frames of the video stream received by the first client device 100 and the remote client device having the visual effects of the self-view of the user of the first client device 100 doubly applied.

[0102] For example, if the first client device 100 is to switch to the first mode (wherein background blur is to be applied to the video stream by the video processing server 300) while the background replacement visual effect is being applied to the video stream, the visual effects mode selector 134 of the first client device 100 may be configured to temporarily operate in a hybrid mode by which background blur is applied to the video stream by the video processing server 300 while also still allowing the background replacement visual effect to be applied to the video stream by the first client device 100 for a predetermined period of time (or until a first keyframe of a remotely modified video stream generated by the video processing server 300 applying background blur to the video stream is received by the first client device 100). After the predetermined period of time expires (or after the first keyframe is received), the visual effects mode selector 134 is configured to disable the background replacement visual effect being applied by the first client device 100, causing the first client device 100 to operate in the first mode. Thus, some frames of the video stream may contain both visual effects for a short period of time, but this approach ensures that unprocessed frames are not forwarded to other remote client devices participating in the video conference session (eg, second client device 100 ′ and third client device 100 ″).

[0103] In an example embodiment, when switching from the second mode to the first mode, the one or more processors 110 of the first client device 100 may be configured to keep displaying the self-view of the user of the first client device 100 through the local stream until the first key frame of the remote modified video stream is received from the video processing server 300. This approach may result in no freezing and minimal frames with double application of effects (e.g., double blurring due to both the first client device 100 and the video processing server 300 applying the visual effect). If the video stream has not been resized for multiple resolutions, the remote views at other client devices (e.g., the second client device 100' and the third client device 100'') in the video conference session may be temporarily frozen when the video processing server 300 is requested by the first client device 100 to apply the visual effect, and once the first key frame is propagated from the first client device 100 to the other client devices through the video processing server 300, the remote views will be unfrozen. The first client device 100 will continue to apply the visual effect until it receives a new modified video stream from the video processing server 300. Therefore, the first client device 100 will receive some frames with the effect applied double. In another example, to make switching from the second mode to the first mode on the self view faster, viewing requests for the self view may be limited to a threshold resolution (eg, 640x360).

[0104] For example, when switching from the first mode to the second mode, the one or more processors 110 of the first client device 100 may be configured to immediately display the self-view using the local stream once the visual effect is applied by the first client device 100, thereby hiding any artifacts. The remote views at other client devices (e.g., the second client device 100' and the third client device 100") in the video conferencing session may display some frames with the effect double-applied (e.g., with double blur) until the visual effect is disabled by the video processing server 300, and may display some pauses / dropped frames while the first client device 100 loads the visual effect.

[0105] In some implementations, in order to reduce freeze time and / or reduce or avoid frames with double effects, other configurations at the first client device 100 and / or the video processing server 300 may be implemented. For example, the video processing server 300 may remain in a state (e.g., a ready state) ready to apply visual effects to the video stream transmitted from the first client device 100 but not actually apply any visual effects until the mode is switched to the first mode. Keeping the video processing server 300 in a state ready to apply visual effects reduces the startup time of the video processing server 300 and reduces the freeze time of remote client devices (e.g., the second client device 100' and the third client device 100'') in the video conference session. In an embodiment, the first client device 100 may be configured to send a timestamp / frame counter to the video processing server 300, indicating the future time when the first client device 100 wants to switch modes. The video processing server 300 may be configured to ensure that the visual effects are enabled / disabled from the frame at the appropriate time. The first client device 100 may be configured to wait until the timestamp / frame counter value and then start / stop applying the visual effects, thereby preventing double effect frames. Depending on the embodiment in which the first client device 100 sends a timestamp / frame counter, the video processing server 300 may be configured to switch modes immediately and / or may remain in a ready state.

[0106] In an embodiment, the first client device 100 may be configured to send metadata about frames of the video stream that provides information about the visual effect to the video processing server 300. When the first client device 100 requests the video processing server 300 to apply a visual effect, the video processing server 300 is configured to generate a modified video stream with the visual effect only when the applied visual effect is different from any visual effect applied to the incoming video stream. According to the embodiment in which the first client device 100 sends metadata about frames of the video stream that provides information about the visual effect to the video processing server 300, the video processing server 300 may be configured to switch modes immediately and / or may remain in a ready state.

[0107] For example, in some implementations, the first client device 100 and / or the video processing server 300 may preload visual effects without actually applying the visual effects to the output stream. When switching from the first mode to the second mode, for example, the first client device 100 may be configured to preload visual effects, request the video processing server 300 to freeze at the current frame with the visual effects applied by the video processing server 300, send a video stream with the visual effects applied by the first client device 100, and then request the video processing server 300 to stop applying the visual effects to the video stream. Therefore, a short video freeze is applied instead of a frame with a double effect. When switching from the second mode to the first mode, for example, the video processing server 300 may be configured to preload visual effects and receive a request from the first client device 100 to freeze the frame with the visual effects applied by the first client device 100. The first client device 100 may stop applying the visual effects to the video stream and send the video stream without the visual effects to the video processing server 300. Then request the video processing server 300 to apply the visual effects to the video stream. Therefore, a short video freeze is applied instead of a frame with a double effect.

[0108] In some implementations, the video conferencing application 130 may have a default setting that causes the first client device 100 to operate in the first mode. That is, the video conferencing application 130 may prefer that the video processing server 300 applies visual effects in the first mode by default during the video conferencing session (rather than the first client device 100 applying visual effects to the video stream in the second mode). In response to the video processing server 300 being unavailable (e.g., due to resource outages, peak usage, bandwidth limitations, etc.), the visual effects mode selector 134 may be configured to switch to the second mode, by which processing of visual effects is applied by the first client device 100. For example, in response to the video processing server 300 running out of graphics computing resources, the video processing server 300 notifies the first client device 100 through a message, through metadata, etc. (e.g., through the first server 200) that the first mode has become unavailable, and the visual effects mode selector 134 will switch to the second mode in which the first client device 100 applies visual effects (i.e., client-side effects). In an example embodiment, if the first client device 100 cannot support the visual effect requested to be applied to the video stream, the one or more processors 110 of the first client device 100 may be configured to control the display 160 of the first client device 100 to display a message indicating that the requested visual effect cannot be applied to the video stream, and the camera 140 of the first client device 100 may be turned off to protect user privacy. In addition, or as an alternative, if the first client device 100 cannot support the visual effect requested to be applied to the video stream, the one or more processors 110 of the first client device 100 may be configured to stop transmitting the video stream. The first client device 100 may notify the user through additional or alternative methods (e.g., through a speaker) that the requested visual effect cannot be applied to the video stream.

[0109] In some implementations, the first client device 100 may be configured to display a self-view of a user of the first client device 100 during a video conference session, wherein the self-view may be generated and displayed locally by the first client device 100, or the self-view may correspond to a video stream that has been modified by the video processing server 300 by applying one or more visual effects to the video stream and looping the modified video stream back to the first client device 100 to be displayed as the user's self-view. For example, when the first client device 100 operates in a first mode and requests the video processing server 300 to apply visual effects to the video stream, the first client device 100 may be configured to receive the modified video stream remotely. Due to network round trips and transcoding performed on the video processing server 300, the remotely received modified video stream may have worse quality and latency than the local self-view generated by the first client device 100. However, according to example embodiments disclosed herein, the quality and latency of the self-view may be improved according to various techniques. Therefore, the media quality and latency of the self-view may be maintained at a level that does not negatively affect the user's perception of the quality of the video conference session. For example, to obtain lower latency (e.g., a threshold of less than one second, a threshold of less than 300 ms, etc.), when sending the modified video stream back to the first client device 100, the video processing server 300 can be configured to enable WebRTC low-latency rendering and set a threshold limit for latency (e.g., by setting a threshold limit in the PlayoutDelayLimits RTP header extension). As an additional or alternative means of improving the quality of the self-view, the video processing server 300 can be configured to adjust the bitrate allocation policy so that sending the modified video stream back to the first client device 100 takes precedence over sending the modified video stream to other client devices (e.g., the second client device 100' and the third client device 100''). That is, the video processing server 300 can be configured to prioritize the looped modified video stream 196 when distributing available bandwidth downstream. Without such prioritization, the user of the first client device 100 may lose his or her self-view when not speaking in a larger meeting, but the user can still be seen by others in the video conference session. As an additional or alternative means of improving the quality of the self-view, one or more processors 110 of the first client device 100 can be configured to reduce the frame rate of a remotely received video stream (e.g., video streams transmitted from other client devices in the video conferencing session such as the second client device 100' and the third client device 100'') as performance adaptation, wherein the modified video stream of the self-view is not included (i.e., maintained at a default frame rate).Despite implementing one or more of the above-described techniques (e.g., by increasing the priority of the modified video stream to be sent back to the first client device 100), if there is still insufficient bandwidth for the looped modified video stream 196, the one or more processors 110 of the first client device 100 may be configured to detect that the video processing server 300 and / or the first server 200 is unable to modify the video stream 190 generated by the first client device 100 using visual effects, and the visual effects mode selector 134 may switch from processing the video in the first mode to processing the video in a second mode in which the first client device 100 modifies the video stream by applying the visual effect to the video stream and generates its own self-view.

[0110] Terms such as "module" and "unit" may be used herein in association with various features of the present disclosure. Such terms may refer to, but are not limited to, software or hardware components or devices that perform certain tasks, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). A module or unit may be configured to reside on an addressable storage medium and to be configured to execute on one or more processors. Thus, by way of example, a module or unit may include components (such as software components, object-oriented software components, class components, and task components), processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcodes, circuit systems, data, databases, data structures, tables, arrays, and variables. The functionality provided in components and modules / units may be combined into fewer components and modules / units or further divided into additional components and modules.

[0111] Aspects of the above-described example embodiments may be recorded in a computer-readable medium (e.g., a non-transitory computer-readable medium) that includes program instructions for implementing various operations embodied by a computer. The medium may also include data files, data structures, etc., alone or in combination with program instructions. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CD ROMs, Blu-ray discs, and DVDs; magneto-optical media, such as optical discs; and other hardware devices specifically configured to store and execute program instructions, such as semiconductor memories, read-only memories (ROMs), random access memories (RAMs), flash memories, USB memories, etc. Examples of program instructions include machine codes (such as, for example, generated by a compiler) and files containing higher-level codes that can be executed by a computer using an interpreter. Program instructions may be executed by one or more processors. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, and vice versa. In addition, non-transitory computer-readable storage media may be distributed between computer systems connected via a network, and computer-readable codes or program instructions may be stored and executed in a decentralized manner. Furthermore, the non-transitory computer-readable storage medium may also be embodied in at least one application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0112] Each frame of the flow chart can represent a unit, module, segment or part of the code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions mentioned in the frame can appear out of order. For example, two frames shown in succession can actually be executed substantially concurrently (simultaneously), or these frames can sometimes be executed in reverse order, depending on the functionality involved.

[0113] Although the present disclosure has been described with respect to various example embodiments, each example is provided by way of explanation, not limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents of such embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the disclosed subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet further embodiments. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

Claims

1. A server system, comprising: one or more memories configured to store instructions; as well as one or more processors configured to execute the instructions stored in the one or more memories to: receiving a video stream associated with the video conferencing session from a first client device, receiving, from the first client device, visual effect information related to one or more visual effects to be applied to the video stream, applying the one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams, and The one or more modified video streams are transmitted to one or more other client devices participating in the video conferencing session.

2. The server system of claim 1, wherein the one or more processors are configured to transmit a modified video stream of the one or more modified video streams to the first client device.

3. A server system as described in claim 2, wherein the one or more processors are configured to: in response to the media quality of the modified video stream transmitted to the first client device being less than a threshold level and / or the latency of the modified video stream transmitted to the first client device being greater than a threshold level, transmit a notification to the first client device, the notification indicating that the first client device will take over control to apply the one or more visual effects to the video stream and / or stop transmitting the one or more modified video streams to the one or more other client devices participating in the video conferencing session.

4. The server system of claim 1, wherein the one or more processors are configured to: generating a plurality of modified video streams based on the received visual effect information, wherein each modified video stream in the plurality of modified video streams has a different resolution, and A modified video stream of the plurality of modified video streams is transmitted to a corresponding client device among the one or more other client devices participating in the video conferencing session based on a data transmission rate between the server system and the corresponding client device.

5. The server system of claim 1, wherein the visual effects information comprises information regarding one or more settings of the one or more visual effects to be applied to the video stream.

6. The server system of claim 5, wherein the one or more settings include a blur radius of a background blur effect to be applied to at least a portion of the video stream.

7. The server system of claim 5, wherein the one or more settings include a brightness level of a lighting effect to be applied to at least a portion of the video stream.

8. The server system of claim 1, wherein the video stream transmitted from the first client device is a self-view of a user of the first client device.

9. The server system of claim 8, wherein in response to the self-view of the user changing to a size greater than a threshold level, the one or more processors are configured to: ceasing to apply the one or more visual effects to the video stream, and A notification is transmitted to the first client device indicating that the first client device is to take over control to apply the one or more visual effects to the video stream and / or to cease transmission of the one or more modified video streams to the one or more other client devices participating in the video conferencing session.

10. A computing device comprising: one or more memories configured to store instructions; as well as one or more processors configured to execute the instructions stored in the one or more memories to: determining whether to process video associated with the video conferencing session in a first mode or a second mode, In response to determining to process the video in the first mode, generating a first video stream and transmitting the first video stream to a server system, and transmitting visual effect information related to one or more visual effects to be applied to the first video stream to the server system, and In response to determining to process the video in the second mode, generating a second video stream by applying the one or more visual effects to the first video stream, and transmitting the second video stream to the server system or a separate server system.

11. A computing device as described in claim 10, wherein the one or more processors are configured to receive a modified video stream from the server system, in which the one or more visual effects have been applied to the first video stream based on the visual effects information transmitted to the server system.

12. A computing device as described in claim 11, wherein the one or more processors are configured to determine to process the video in the second mode and / or stop transmitting the first video stream to the server system in response to the media quality of the modified video stream being less than a threshold level and / or the latency of the modified video stream received from the server system being greater than a threshold level.

13. The computing device of claim 10, wherein the visual effects information comprises information related to one or more settings of a visual effect to be applied to the first video stream.

14. The computing device of claim 13, wherein the one or more settings include a blur radius for a background blur effect to be applied to at least a portion of the first video stream.

15. The computing device of claim 13, wherein the one or more settings include a brightness level of a lighting effect to be applied to at least a portion of the first video stream.

16. The computing device of claim 10, wherein The video includes a self-view of a user of the computing device, and The one or more processors are configured to determine whether to process the video in the first mode or the second mode based on a size of the self-view.

17. The computing device of claim 16, wherein When the size of the self-view is less than a threshold level, the one or more processors are configured to determine to process the video in the first mode, and When the size of the self-view is greater than the threshold level, the one or more processors are configured to determine to process the video in the second mode.

18. The computing device of claim 10, wherein The video includes a plurality of self-views of a user of the computing device, The one or more processors are configured to determine a self-view having a largest size among the plurality of self-views, and The one or more processors are configured to determine whether to process the video in the first mode or the second mode based on a size of the self-view having the maximum size.

19. A computing device as described in claim 10, wherein the visual effect information includes instructions for the server system to not transmit the first video stream to other client devices participating in the video conferencing session if the one or more visual effects cannot be applied to the first video stream.

20. A computer-implemented method comprising: determining whether to process video associated with the video conferencing session in a first mode or a second mode; In response to determining to process the video in the first mode, generating a first video stream and transmitting the first video stream to a server system, and transmitting visual effect information related to one or more visual effects to be applied to the first video stream to the server system; as well as In response to determining to process the video in the second mode, generating a second video stream by applying the one or more visual effects to the first video stream, and transmitting the second video stream to the server system or a separate server system.

Images