Cloud-based application of visual effects to videos

Abstract

A server system (300) receives from a first client device (100) a video stream (190, 190') for the video conferencing session, and receives from the first client device visual effects information for one or more visual effects to apply to the video stream, the server system applies one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams (192, 194), and transmits the one or more modified video streams to one or more other client devices participating in the video conferencing session.

Description

[Technical field]

[0001] Claiming priority This application claims priority to U.S. patent application Ser. No. 17 / 738,176, filed May 6, 2022, which is incorporated by reference in its entirety herein.

[0002] The present disclosure relates to video processing for video telephony (which may also be referred to as video calling or video conferencing) conducted over a network. More specifically, embodiments disclosed herein relate to video processing operations for video conferencing that are performed in the cloud by one or more servers, rather than by client devices. [Background technology]

[0003] Video telephony comprises a remote communication system that uses computing devices to transmit audio and visual signals, allowing multiple people from different locations to participate in a conversation in real time.

[0004] Some video telephony applications support various visual effects, such as background blurring and light and dark adjustments, etc. However, not all client devices (e.g., laptops, tablets, smartphones, etc.) are powerful enough to render these visual effects without degrading the performance of the client device and / or the video conferencing application. Summary of the Invention

[0005] Aspects and advantages of embodiments of the disclosure are set forth in part in the description that follows, and may be learned from the description, or may be learned by practice of exemplary embodiments.

[0006] In one or more exemplary embodiments, a server system (e.g., a video processing server) comprises one or more memories configured to store instructions and one or more processors configured to execute the instructions stored in the one or more memories to perform the following operations: receive a video stream related to the video conferencing session from a first client device; receive visual effects information from the first client device related to one or more visual effects to apply to the video stream; apply one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams; and transmit the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0007] In some implementations, the one or more processors of the server system are configured to transmit one of the one or more modified video streams to the first client device.

[0008] In some implementations, the one or more processors of the server system are configured to, in response to a media quality of the modified video stream sent to the first client device being below a threshold level and / or a latency of the modified video stream sent to the first client device exceeding a threshold level, send a notification to the first client device indicating that the first client device is assuming control for applying one or more visual effects to the video stream and / or to stop sending the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0009] In some implementations, the one or more processors of the server system are configured to generate a plurality of modified video streams, each having a different resolution, based on the received visual effects information, and to transmit one of the plurality of modified video streams to each of the one or more other client devices participating in the video conferencing session based on a data transfer rate between the server system and each of the client devices.

[0010] In some implementations, the visual effects information includes information regarding one or more settings of one or more visual effects to apply to the video stream. In some implementations, the one or more settings include a blur radius of a background blur effect to apply to at least a portion of the video stream. In some implementations, the one or more settings include a brightness level of a lighting effect to apply to at least a portion of the video stream.

[0011] In some implementations, the video stream transmitted from the first client device is a self-view of a user of the first client device. In some implementations, in response to the user's self-view being resized above a threshold level, the one or more processors of the server system are configured to: stop applying one or more visual effects to the video stream; and send a notification to the first client device indicating that the first client device assumes control for applying the one or more visual effects to the video stream and / or to stop transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0012] In one or more exemplary embodiments, a computing device (e.g., a client device) comprises one or more memories configured to store instructions and one or more processors configured to execute the instructions stored in the one or more memories, the one or more processors executing the instructions to: determine whether to process video for a video conferencing session in a first mode or a second mode; in response to determining to process the video in the first mode, generate a first video stream, transmit the first video stream to a server system, and transmit visual effects information to the server system regarding one or more visual effects to apply to the first video stream; and in response to determining to process the video in the second mode, generate a second video stream by applying one or more visual effects to the first video stream and transmit the second video stream to the server system or a separate server system.

[0013] In some implementations, the one or more processors of the computing device are configured to receive from the server system a modified video stream in which one or more visual effects have been applied to the first video stream based on the visual effects information transmitted to the server system.

[0014] In some implementations, the one or more processors of the computing device are configured to determine to process the video in a second mode and / or to stop transmitting the first video stream to the server system in response to the media quality of the modified video stream being below a threshold level and / or the latency of the modified video stream received from the server system exceeding a threshold level.

[0015] In some implementations, the visual effects information includes information regarding one or more settings of a visual effect to apply to the first video stream. In some implementations, the one or more settings include a blur radius of a background blur effect to apply to at least a portion of the first video stream. In some implementations, the one or more settings include a brightness level of a lighting effect to apply to at least a portion of the first video stream.

[0016] In some implementations, the video includes a self-view of a user of the computing device, and one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on a size of the self-view. In some implementations, when a size of the self-view is below a threshold level, the one or more processors of the computing device are configured to determine to process the video in the first mode, and when a size of the self-view exceeds the threshold level, the one or more processors of the computing device are configured to determine to process the video in the second mode.

[0017] In some implementations, the video includes multiple self-views of a user of the computing device, and one or more processors of the computing device are configured to identify a self-view having a maximum size from among the multiple self-views, and the one or more processors of the computing device are configured to determine whether to process the video in a first mode or a second mode based on the size of the self-view having the maximum size.

[0018] In some implementations, the visual effects information includes instructions for the server system not to transmit the first video stream to other client devices participating in the video conferencing session if one or more visual effects cannot be applied to the first video stream.

[0019] In one or more exemplary embodiments, a computer-implemented method for a computing device (e.g., a client device) includes determining whether to process video for a video conferencing session in a first mode or a second mode; in response to determining to process the video in the first mode, generating a first video stream, transmitting the first video stream to a server system, and transmitting visual effects information to the server system regarding one or more visual effects to apply to the first video stream; and in response to determining to process the video in the second mode, generating a second video stream by applying one or more visual effects to the first video stream and transmitting the second video stream to the server system or a separate server system.

[0020] In one or more exemplary embodiments, a computer-implemented method for a server system (e.g., a video processing server) includes receiving a video stream for a video conferencing session from a first client device; receiving visual effects information from the first client device regarding one or more visual effects to apply to the video stream; applying the one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams; and transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0021] In one or more exemplary embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions executable by one or more processors of a client device and / or a server system. In some implementations, the computer-readable medium stores instructions, which may include instructions that cause one or more processors to perform one or more operations (e.g., server system operations and / or client device operations) of any of the methods described herein. The computer-readable medium may store additional instructions for performing other aspects of the server system and client device and corresponding methods of operation as described herein.

[0022] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description, drawings, and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the relevant principles.

[0023] A detailed discussion of exemplary embodiments directed to persons skilled in the art is set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0024] [Figure 1] 1 illustrates an exemplary video conferencing system, in accordance with one or more exemplary embodiments of the present disclosure. [Figure 2A] 1 illustrates an example flow diagram for a video processing server, in accordance with one or more example embodiments of the present disclosure. [Figure 2B] 1 illustrates an example flow diagram for a video processing server, in accordance with one or more example embodiments of the present disclosure. [Diagram 3] 1 illustrates an example block diagram of a client device and a video processing server in accordance with one or more example embodiments of the present disclosure. [Figure 4]1 illustrates an example display of a video conferencing session on a client device, in accordance with one or more example embodiments of the present disclosure. [Diagram 5] 1 illustrates an example display of visual effects applied to a user's self-view during a video conferencing session at a client device. [Figure 6] 1 illustrates a flow diagram of an exemplary and non-limiting computer-implemented method in accordance with one or more exemplary embodiments of the present disclosure. [Figure 7] 1 illustrates a flow diagram of an exemplary and non-limiting computer-implemented method in accordance with one or more exemplary embodiments of the present disclosure. [Figure 8] 1 illustrates a flow diagram of an exemplary and non-limiting computer-implemented method in accordance with one or more exemplary embodiments of the present disclosure. [Figure 9] 1 illustrates a flow diagram of an exemplary and non-limiting computer-implemented method in accordance with one or more exemplary embodiments of the present disclosure. [Figure 10] 1 illustrates a flow diagram of an exemplary and non-limiting computer-implemented method in accordance with one or more exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Reference will now be made to embodiments of the present disclosure, one or more examples of which are illustrated in the drawings, in which like reference numerals refer to like elements. Each example is provided as an explanation of the disclosure, and is not intended to limit the disclosure. Indeed, it will be apparent to those skilled in the art that various modifications and variations may be made to the present disclosure without departing from the scope or spirit of the disclosure. For example, features illustrated or described as part of one embodiment can be used with another embodiment to yield yet another embodiment. Thus, the present disclosure is intended to cover such modifications and variations as come within the scope of the appended claims and their equivalents.

[0026] The terms used herein are used to describe exemplary embodiments and are not intended to limit and / or restrict the present disclosure. The singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly dictates otherwise. In the present disclosure, terms such as "including", "having" and "comprising" are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more of the features, numbers, steps, operations, elements, components, or combinations thereof.

[0027] Terms such as first, second, third, etc. may be used herein to describe various elements, but it will be understood that these terms are not intended to limit the elements. Instead, these terms are used to distinguish one element from another. For example, a first element may be referred to as a second element, and a second element may be referred to as a first element, without departing from the scope of the present disclosure.

[0028] When an element is referred to as "connected" to another element, it will be understood that this expression encompasses examples of direct connections or couplings, as well as intervening connections or couplings between the elements with one or more other elements.

[0029] The term "and / or" includes a combination of multiple associated listed items or any of multiple associated listed items. For example, the scope of an expression or phrase "A and / or B" includes item "A," item "B," and the combination of items "A and B."

[0030] Additionally, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0031] According to an example embodiment, aspects of the video processing for video conferencing may be performed on the cloud by one or more servers rather than by the client devices, thus relieving the client devices of processing tasks (e.g., central processing unit / graphics processing unit (CPU / GPU) tasks) that may cause the client devices to overheat, degrade display quality, or otherwise degrade performance. According to an example embodiment, the one or more servers receive video streams for the video conferencing session from the client devices. The one or more servers also receive visual effects information from the client devices regarding one or more visual effects to apply to the video streams. The one or more servers apply one or more visual effects to the video streams based on the received visual effects information to generate one or more modified video streams, and transmit the one or more modified video streams to one or more other client devices participating in the video conferencing session.

[0032] According to an embodiment of the present disclosure, a video stream is obtained from a client device, and media features corresponding to visual effects to be added to the video stream are processed in the cloud by one or more servers, hereinafter referred to as video processing servers. A video processing server is a server (or a combination of servers) configured to receive a video stream from a client device (possibly via an intermediate server) and transmit the video stream (possibly in a modified form) to one or more other client devices. In an exemplary embodiment, the video processing server may modify the video stream by adding one or more visual effects to the video stream based on information received from the client device regarding one or more visual effects to apply to the video stream. The video processing server may also change the resolution of the modified or unmodified video stream based on the requirements of the respective client device receiving the modified or unmodified video stream. Thus, the video processing server has an input video stream from a client device and has multiple outputs corresponding to multiple modified or unmodified video streams with different resolutions according to the requirements of the respective receiving client device. In an exemplary embodiment, when the video stream is modified by the video processing server, the modified video stream is sent to the client device that sent the video stream to the cloud in addition to being sent to other client devices participating in the video conference, so that the user of the client device can also view the modified video stream with visual effects applied to the video stream during the video conference.

[0033] According to an exemplary embodiment, a client device determines whether to process a video for a video conference session in a first mode or a second mode. In response to determining to process the video in the first mode, the client device generates a first video stream, transmits the first video stream to a server, and transmits visual effect information to the server regarding a visual effect to apply to the first video stream. In response to determining to process the video in the second mode, the client device generates a second video stream by applying a visual effect to the first video stream, and transmits the second video stream to the server (or another server). For example, in the first mode, the client device transmits a raw video stream without visual effects to the server, and further transmits visual effect information regarding a visual effect to apply to the first video stream. For example, the visual effect may be a blur effect to apply to a background of the self-view video stream. The server receives the raw video stream, and applies a blur effect to a background of the self-view video stream based on the visual effect information to generate a modified video stream. The server transmits the modified video stream to other client devices participating in the video conferencing session, and possibly also transmits the modified video stream to the client device that sent the raw video stream to the server.

[0034] According to embodiments disclosed herein, visual effects may be applied to the video stream by one or more servers, referred to as video processing servers as described above. For example, a video processing server may receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to the uncompressed media. To implement visual effects on the video stream, the video processing server may utilize libraries that support combining visual rendering effects (e.g., blurring, scaling, lighting, etc.) with performing foreground segmentation (e.g., segmenting a user and applying visual effects to the background in the self-view of the user of the client device). The video processing server may be specially configured to perform video processing on the video stream received from the client device to apply visual effects to the video stream. For example, the video processing server may include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or applying visual effects to the video stream), and multiple video encoder accelerators (e.g., application specific integrated circuits (ASICs) for performing transcoding operations such as encoding, decoding, and resizing).

[0035] According to embodiments disclosed herein, a loopback stream is implemented from the video processing server to the client device so that a user of the client device can see the visual effects applied to the video stream. In an exemplary embodiment, one or more intermediate servers may be disposed between the client device and the video processing server, such that the one or more intermediate servers route the video stream from the client device to the video processing server, and the one or more intermediate servers route the modified video stream from the video processing server to the client device.

[0036] The video processing server is configured to perform video transcoding operations. For example, the video processing server may receive compressed media (e.g., video) from a client device, decompress the media, and apply visual effects to the uncompressed media. To implement visual effects on a video stream, the video processing server may utilize a library that supports combining visual rendering effects (e.g., blurring, scaling, lighting, etc.) with performing foreground segmentation (e.g., segmenting a user and applying visual effects to the background in the user's self-view of a client device).

[0037] For example, when a client device applies visual effects to the video stream itself, the video processing server may be bypassed and other servers in the computing network (server system) may be utilized to transmit the video stream to the client device participating in the video conference session. In another exemplary embodiment, when the video processing server is not required or necessary to apply visual effects to the input video stream, the video processing server is configured to decode the video stream received from the client device and pass the decoded video stream to a resizer node, i.e., multiple resizer nodes, one for each unique resolution requested by each client device receiving the video stream of the video conference session. That is, the video processing server is configured to perform a video transcoding operation using a graph of nodes. In some implementations where visual effects are not applied by the video processing server, the video stream does not pass through the video processing server since there is no need for a resizing operation by the video processing server, and a client device requesting a highest resolution video stream may receive the video stream from the client device via another server. According to another embodiment, the video processing server is configured to pass the decoded video stream to multiple resizer nodes, each of which is bucketed to correspond to a particular bitrate resolution of each client device participating in the video conferencing session, which may or may not correspond exactly to the resolution of each client device. For example, each client device may have the capability to process a high definition (HD) video stream (e.g., 1920x1080) but may be limited to a particular resolution (e.g., 800x600) due to bitrate and / or bandwidth constraints, so the video processing server may resize the modified video stream according to the particular resolution based on the bitrate of each client device.

[0038] According to an example embodiment disclosed herein, a client device is configured to request a video processing server to apply one or more visual effects to a video stream that the client device has generated and sent to the video processing server (e.g., via an intermediate server). The client device may be configured to signal the video processing server to apply the visual effects to the video stream, for example, in configuration properties associated with the video stream (e.g., via metadata). The configuration properties may include one or more settings of the one or more visual effects to be applied to the video stream by the video processing server. When the client device requests the video processing server to apply the visual effects to the video stream, the video processing server is configured to decode the video stream received from the client device and pass the decoded video stream to a visual effects applier (also referred to as a media pipe effects module) to apply the visual effects to the video stream to generate a modified video stream. The modified video stream is then passed to a resizer node, thus multiple resizer nodes, one for each unique resolution requested by each client device that receives the modified video stream. According to another embodiment, the video processing server is configured to pass the modified video stream to multiple resizer nodes, each of which is bucketed to correspond to a particular bitrate resolution of each client device participating in the video conferencing session, which may or may not correspond exactly to the resolution of each client device.

[0039] According to exemplary embodiments disclosed herein, the client device may be configured to receive the modified video stream from the video processing server. The client device may specifically request that the video processing server return the modified video stream to the client device, or the video processing server may be configured to automatically return the modified video stream to the client device. For example, when a visual effect is requested by the client device, the video processing server is configured to signal (e.g., via a message or via metadata) to an intermediate server (i.e., a server located between the client device and the video processing server, which forwards the video stream received from the client device to the video processing server) that the video stream received from the client device is at least partially used to be sent from the video processing server to the client device via the intermediate server as well. Thus, in response to receiving the signal from the video processing server, the intermediate server prepares to return at least a portion of the video stream to the client device, and a push notification is sent to the client device informing the client device about the new stream. When the client device receives the notification about the modified stream, the client device is configured to perform a search for input data from the intermediate server, including the modified video stream. The client device may be configured to display the modified video stream in preference to a local stream generated at the client device.

[0040] According to example embodiments disclosed herein, a client device may be configured to toggle or switch between a first mode in which the client device requests that the video processing server apply visual effects to a video stream generated by the client device, and a second mode in which the client device applies visual effects to the video stream and sends the modified video stream to the video processing server. For example, if a user of the client device switches to a first mode in which the video processing server applies background blur to the video stream while a background replacement visual effect is being applied to the video stream, one or more processors of the client device may be configured to first enable the first mode, which still allows the video processing server to apply background blur to the video stream while simultaneously temporarily applying the background replacement visual effect until the background replacement visual effect is disabled. Thus, although some frames of the video stream may include both visual effects, such an approach ensures that unprocessed frames are not forwarded to other client devices participating in the video conference session.

[0041] According to example embodiments disclosed herein, a client device may be configured to distinguish between a "waiting room" or "waiting room" state, in which a user is waiting to be admitted or join a video conference meeting, and a "joining" state, in which a user has been admitted or joined the meeting, when switching between a first mode and a second mode. For example, in response to a client device transitioning from a waiting room to a joining state, one or more processors of the client device may be configured to switch from the second mode to the first mode, such that the video processing server is responsible for modifying the video stream generated by the client device with visual effects. For example, a background blur effect (or other visual effect) applied by the client device to the video stream while the client device was in the waiting room waiting to be admitted to the meeting "transfers" from the client to the video processing server when the client device is admitted to the meeting. Thus, when the client device joins the meeting, the video processing server is responsible for applying the background blur effect (or other visual effect).

[0042] According to the exemplary embodiments disclosed herein, a video conferencing application may have a first mode set by default. That is, by default, the video conferencing application may prefer that the video processing server apply visual effects in a first mode (rather than in a client device of a second mode) during a video conferencing session. In response to the video processing server being unavailable (e.g., due to resource shortage, high peak usage, bandwidth limitation, etc.), one or more processors of the client device may be configured to switch to a second mode, whereby the processing of visual effects is applied by the client device. For example, in response to a lack of graphic computing resources in the video processing server, the intermediate server is configured to signal back to the client device (e.g., via a message or via metadata) that the first mode (i.e., the visual effects are applied by the video processing server rather than by the client device) is unavailable, and the client device falls back to a second mode in which the client device applies visual effects (i.e., client-side effects). In an exemplary embodiment, if the client device cannot support the visual effects required to be applied to the video stream, one or more processors of the client device may be configured to control the display of the client device to display a message indicating that the required visual effects cannot be applied to the video stream, and the camera of the client device is muted to protect the privacy of the user. The client device may notify the user in an additional or alternative way (e.g., via a speaker) that the required visual effects cannot be applied to the video stream.

[0043] In an exemplary embodiment, if a client device requests that a video processing server apply a visual effect to a video stream and the video processing server is unable to modify the video stream (e.g., due to lack of resources or processing limitations), the video processing server may be configured to forward the video stream to other client devices in the video conference session without applying the visual effect. However, in an exemplary embodiment, if the video processing server is unable to modify the video stream (e.g., due to lack of resources or processing limitations), the video processing server may also be configured to not forward the video stream to other client devices in the video conference session. If the video processing server is unable to modify the video stream as requested, the video processing server may receive instructions from the client device not to forward the video stream to other client devices in the video conference session. This aspect provides additional privacy to the user of the client device if the requested visual effect is privacy related (e.g., if the user does not want other users in the video conference session to see the unblurred background). Thus, remote client devices in the video conference session may receive the modified video stream or may not receive any video at all during the intermittent outage.

[0044] According to exemplary embodiments disclosed herein, a client device may be configured to display a self-view of a user of the client device during a video conferencing session. For example, when the client device operates in a first mode and requests that a video processing server apply visual effects to the video stream, the client device may be configured to receive a modified video stream "remotely." The modified video stream received remotely may have poorer quality and latency compared to a local self-view generated by the client device due to network round trips and transcoding performed by the video processing server. However, according to exemplary embodiments disclosed herein, the quality and latency of the self-view may be improved according to various techniques. Thus, the media quality and latency of the self-view may be maintained at a level that does not adversely affect the user's perception of the quality of the video conferencing session. For example, to obtain lower latency (e.g., sub-1 second threshold, sub-300 mms threshold, etc.), the video processing server may be configured to enable WebRTC low latency rendering and set a latency threshold limit when sending the modified video stream back to the client device (e.g., by setting the threshold limit in the PlayoutDelayLimits RTP header extension). As an additional or alternative approach to improving self-view quality, the video processing server may be configured to adjust the bitrate allocation strategy to prioritize sending the modified video stream back to the client device over sending the modified video stream to other client devices. That is, the video processing server may be configured to prioritize the modified video stream that is looped back when distributing available bandwidth downstream. Without this prioritization, in larger meetings, the user of the client device loses self-view when not speaking, but other users in the video conferencing session can still see the user.As an additional or alternative approach to improving self-view quality, one or more processors of the client device may be configured to reduce the frame rate of remotely received video streams (e.g., video streams sent from other client devices in the video conference session) as a performance adaptation, but excluding the modified video stream of the self-view (i.e., maintained at a default frame rate). If, despite one or more of the aforementioned techniques being implemented (e.g., by increasing the priority of the modified video stream to send back to the client device), there is still insufficient bandwidth for the modified video stream to be looped back, one or more processors of the client device may be configured to detect that the video processing server and / or intermediate server cannot modify the video stream generated by the client device with visual effects, and the client device may switch from processing the video in a first mode to processing the video in a second mode, and the client device modifies the video stream by applying visual effects to the video stream.

[0045] According to an exemplary embodiment disclosed herein, a client device may be configured to perform a toggle process or a switch between a first mode in which the client device requests that a video processing server apply a visual effect to a video stream generated by the client device based on the size of the self-view and / or the physical resolution of the display of the client device, and a second mode in which the client device applies a visual effect to the video stream and transmits the modified video stream to the video processing server. The size of the self-view may correspond to the active resolution of the self-view. For example, during a video conference session, when the size of the self-view displayed on the display of the client device exceeds a threshold, one or more processors of the client device are configured to switch (or maintain the second mode) to the second mode in which the client device applies a visual effect to the video stream and transmits the modified video stream to the video processing server. When the size of the self-view exceeds the threshold, video quality problems perceptible to the user may occur on the client device. Conversely, when the size of the self-view displayed on the display of the client device is less than the threshold, one or more processors of the client device are configured to switch (or maintain the first mode) to the first mode in which the client device requests that the video processing server apply a visual effect to the video stream generated by the client device.

[0046] In an embodiment, when multiple self-views are displayed on the display of the client device during a video conference session, the one or more processors of the client device are configured to identify a self-view with a maximum size. The one or more processors of the client device are configured to determine whether a size of the self-view with the maximum size exceeds a threshold to determine whether the client device switches to a second mode (or remains in the second mode) in which the client device applies visual effects to the video stream and sends the modified video stream to the video processing server. The size of the self-view may correspond to an active resolution of the self-view with the maximum size. The one or more processors may be configured to track or monitor the size of the one or more self-views during the video conference session to determine whether there is a change in the maximum size and whether the client device switches to a second mode (or remains in the second mode) in which the client device applies visual effects to the video stream and sends the modified video stream to the video processing server. Changes regarding the size and / or position of the self-view may be transmitted to the video processing server to update the position of the visual effects applied to the self-view depending on whether the client device switches to the first mode or the second mode. In an embodiment, when the size of the self-view is frequently resized by a user, the one or more processors may be configured to wait until the resizing has stopped for a predetermined time (e.g., 3 seconds) before determining whether the size of the self-view displayed on the display of the client device exceeds a threshold.

[0047] When a client device toggles or switches between a first mode in which the client device requests that the video processing server apply visual effects to a video stream generated by the client device, and a second mode in which the client device applies visual effects to the video stream and sends the modified video stream to the video processing server, intermittent video problems (such as video freezing, frame drops, etc.) may occur. Such intermittent video problems may be more evident in the self-view than in the view of a remote participant of the video conference session. In an exemplary embodiment, when switching from one implementation to another (i.e., from the first mode to the second mode, or vice versa), the client device is configured to wait until the new implementation is started (i.e., until the client device starts receiving remote frames of the self-view generated by the video processing server or until visual effects are applied by the client device) before shutting down the previous implementation.

[0048] For example, when switching from the second mode to the first mode, one or more processors of the client device may be configured to continue displaying the self-view via the local stream until the first key frame of the remote modified video stream is received from the video processing server. This approach may result in no freezing and a minimal number of double effected frames (e.g., double blurred frames with visual effects applied by both the client device and the video processing server). If the video stream has not yet been resized for multiple resolutions, the remote view at other client devices in the video conference session may freeze when the visual effects are requested from the video processing server and recover from the freezing when the first key frame is propagated from the client device through the video processing server. The client device continues to apply visual effects until it sees the new modified video stream, resulting in some double effected frames. Both the client device participating in the video conference session and the remote client device may see quality degradation due to the restart of the video stream.

[0049] For example, when switching from a first mode to a second mode, one or more processors of the client device may be configured to display the self-view using the local stream as soon as the client applies the visual effects without any perceptible artifacts. Remote views at other client devices in the video conference session may display some frames with double effects applied (e.g., double blur) and may display some stalls / frame drops as the client device loads the visual effects until disabling of the visual effects by the video processing server is performed.

[0050] In some implementations, other configurations in the client device and / or video processing server may be implemented to reduce the freezing time and / or reduce or avoid double effect frames. For example, the video processing server may be maintained in a state ready to apply visual effects (e.g., in a ready state) to the video stream sent from the client device, but not actually apply any visual effects until the mode is switched to the first mode. By maintaining the video processing server in a state ready to apply visual effects, the start-up time of the video processing server is reduced and the freezing time of the remote client device during the video conference session is reduced. In an embodiment, the client device may be configured to send a timestamp / frame counter to the video processing server indicating a future time at which the client device would like to switch modes. The video processing server is configured to ensure that the visual effects are enabled / disabled in that frame at the appropriate time. The client device is configured to wait until the value of the timestamp / frame counter before starting / stopping the application of the visual effects, preventing double effect frames. According to this embodiment in which the client device sends the timestamp / frame counter, the video processing server may be configured to switch modes immediately and / or may be maintained in a ready state.

[0051] In an embodiment, a client device may be configured to send metadata for frames of a video stream, the metadata providing information about visual effects to the video processing server. When a client device requests the video processing server to apply a visual effect, the video processing server is configured to generate a modified video stream with the visual effect only if the applied visual effect is different from any visual effect applied to the input video stream. According to this embodiment in which a client device sends metadata for frames of a video stream that provides information about visual effects to the video processing server, the video processing server may be configured to switch modes on the fly and / or remain in a ready state.

[0052] For example, in some implementations, the client device and / or the video processing server may preload the visual effects without actually applying the visual effects to the output stream. For example, when switching from a first mode to a second mode, the client device may be configured to preload the visual effects, request the video processing server to freeze at the current frame on which the visual effects have been applied by the video processing server, send the video stream on which the visual effects have been applied by the client device, and then request the video processing server to stop applying the visual effects to the video stream. Thus, a short video freeze is applied instead of a double effect frame. For example, when switching from a second mode to a first mode, the video processing server may be configured to receive a request from the client device to preload the visual effects and freeze the frame on which the visual effects have been applied by the client device. The client device may stop applying the visual effects to the video stream and send the video stream without the visual effects to the video processing server. The video processing server is then requested to apply the visual effects to the video stream. Thus, a short video freeze is applied instead of a double effect frame.

[0053] In an exemplary embodiment, a mixed mode of operation may be implemented between the client device and the video processing server. For example, when the client device switches to apply visual effects to the video stream, one or more processors of the client device may be configured to send an unmodified video stream to the video processing server while simultaneously displaying a self-view with the video stream with the visual effects applied by the client device, so that the video processing server continues to apply visual effects to the unmodified video stream and the modified video stream (with the visual effects applied by the video processing server) is sent to be displayed at the remote client device participating in the video conference session. In this approach, the switch from the first mode is completely invisible to the remote client device, but the self-view at the client device may experience temporary glitches when switching to the mixed mode due to different timelines of the video streams. In an implementation, to avoid running in the mixed mode for a long time, the client device may be configured to switch from the mixed mode to the second mode after a certain predetermined time. This approach prevents the waste of cloud resources when the self-view is determined to be below a threshold size, thus allowing the client device to apply visual effects without degrading the self-view too much.

[0054] In another exemplary embodiment, to make switching from the second mode to the first mode in the self-view faster, the view request for the self-view may be limited to a threshold resolution (e.g., 640x360).

[0055] In another exemplary embodiment, when any zoom function (e.g., auto-zoom or centering function) is enabled in the video conferencing application at the client device, switching to the first mode may be disabled. However, in an embodiment, when the zoom function is enabled in the video conferencing application at the client device, the video processing server may be configured to apply visual effects to the video stream in the first mode. For example, a magnified stream (i.e., a raw video stream modified with zoom at the client) may be sent from the client device to the video processing server, and visual effects may be applied to the magnified stream. For example, a raw video stream may be sent from the client device to the video processing server, and the video processing server may apply both the magnification and visual effects to the video stream.

[0056] The present disclosure provides numerous technical effects and advantages. Video processing is a costly processing task (e.g., CPU / GPU intensive task) that can consume significant bandwidth of client devices such as laptops, tablets, and smartphones. Adding visual effects to a video stream while running a video conferencing application can cause the client device to overheat, drain battery power, and / or the client device may struggle to implement high quality visual effects (e.g., reduce frames per second or slow down the client device). As an example of technical effects and advantages, the systems and methods disclosed herein enable certain video processing to be offloaded to the cloud (e.g., to a server such as a video processing server), thereby relieving pressure on the client device. Thus, by providing a server such as a video processing server configured with hardware that modifies the video stream sent by the client device to include the visual effects requested by the client device, it is possible to facilitate high quality and efficient video conferencing on the client device. Technical advantages include reduced client CPU usage, improved stability, increased bandwidth, improved video quality (e.g., higher resolution and / or frame rate), and improved user experience.

[0057] Referring now to the drawings, FIG. 1 is an example video conferencing system according to one or more example embodiments of the present disclosure. FIG. 1 illustrates an example of a video conferencing system including a first client device 100, a second client device 100′, a third client device 100″, a first server (first intermediate server) 200, a second server (second intermediate server) 200′, a third server (third intermediate server) 200″, and a video processing server 300, each of which may communicate with each other via a network. For example, the first client device 100, the second client device 100′, and the third client device 100″ may include any of a personal computer, a smartphone, a laptop, a tablet computer, and the like. The network may include any type of communication network, such as a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a personal area network (PAN), or a virtual private network (VPN). For example, wireless communication between elements of the exemplary embodiments may be performed via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Radio Frequency (RF) signals, etc. For example, wired communication between elements of the exemplary embodiments may be performed via cable pairs, coaxial cables, fiber optic cables, Ethernet cables, etc.

[0058] As described in more detail below, in some implementations, the first client device 100 and one or more other client devices (e.g., the second client device 100', the third client device 100', etc.) may participate in a video telephony (hereinafter referred to as video conferencing) session using the video processing server 300 over a network. A video conferencing session may include multiple client devices, including client devices other than the first client device 100, the second client device 100', and the third client device 100'' shown in FIG.

[0059] In FIG. 1 , the first client device 100 transmits a video stream 190 (e.g., a raw video stream) to the video processing server 300 via the first server 200. The video stream 190 may correspond to a self-view of a user of the first client device 100 who is participating or waiting to participate in a video conferencing session with other client devices. In an embodiment, the video stream 190 may be a raw video stream with no visual effects applied to the video stream (e.g., no background blur and / or lighting effects applied to the user's self-view). In addition to the video stream 190, in an embodiment, the first client device 100 transmits visual effects information 190′ (e.g., media features) to the video processing server 300 via the first server 200, the visual effects information corresponding to visual effects applied to the video stream by the video processing server 300.

[0060] In response to receiving the video stream 190 and the visual effects information 190', the video processing server 300 modifies the video stream 190 by adding one or more visual effects to the video stream 190 based on the visual effects information 190'. The video processing server 300 transmits the modified video stream to other client devices participating in the video conferencing session. For example, the video processing server 300 transmits the modified video stream 192 to the second client device 100' via the second server 200' and transmits the modified video stream 194 to the third client device 100'' via the third server 200''. In some implementations, the video processing server 300 may transmit the modified video stream (loopback stream) 196 back to the first client device 100 via the first server 200. In some implementations, the modified video streams 192, 194, 196 may be the same (e.g., have the same resolution), while in other implementations, the modified video streams 192, 194, 196 may be different from one another (e.g., have different resolutions). Further details regarding the operation of the video processing server 300 are discussed below.

[0061] 2A-2B show an example flow diagram for a video processing server according to one or more example embodiments of the present disclosure. For example, when the first client device 100 applies visual effects to the video stream by itself, the video processing server 300 may be bypassed and other servers in the computing network (server system) may be utilized to transmit the video stream to the client devices participating in the video conference session. In another example embodiment, when the video processing server 300 is not required or necessary to apply visual effects to the input video stream, the video processing server 300 is configured to perform the following: decoding the video stream received from the first client device 100, and passing the decoded video stream to a resizer node (resizer), i.e., multiple resizer nodes (resizers), one for each unique resolution requested by each client device receiving the video stream of the video conference session. That is, the video processing server 300 is configured to perform a video transcoding operation using a graph of nodes. Transcoding refers to operations such as decompressing (e.g., decoding) a video stream, modifying the decompressed video stream (e.g., applying one or more visual effects, resizing the video stream, etc.), and recompressing (e.g., encoding) the video stream. In implementations where visual effects are not applied by the video processing server 300, the video stream does not pass through the video processing server 300 (because no resizing operations are required by the video processing server 300), and a client device requesting a full resolution video stream may receive the video stream from the client device via another server. According to another embodiment, the video processing server is configured to pass the decoded video stream to multiple resizer nodes, each of which is bucketed to correspond to a particular bitrate resolution of each client device participating in the video conferencing session, which may or may not correspond exactly to the resolution of each client device.For example, each client device may have the capability to process high definition (HD) video streams (e.g., 1920x1080) but may be limited to a particular resolution (e.g., 800x600, 640x360, etc.) due to bitrate and / or bandwidth constraints, so the video processing server 300 may resize the video streams according to the particular resolution based on the bitrate of each client device.

[0062] 2A, the video processing server 300 may be configured to process an input video stream 210 received from a first client device 100, where the input video stream 210 has had visual effects applied to it at the client side (i.e., by the first client device 100). In the embodiment of FIG. 2A, the video processing server 300 processes the input video stream 210 by a decoder 332 that decodes the input video stream 210. The video processing server 300 is also configured to resize the decoded video stream by resizers 334a, 334b ​​that perform resizing operations to obtain a desired resolution of the input video stream 210. As mentioned above, the desired resolution may be based on the capabilities of the client device receiving the video stream from the video processing server 300, or may be based on existing bitrate and / or bandwidth constraints for the client device receiving the video stream from the video processing server 300. For example, the resizer 334a may be configured to resize the decoded video stream at a first resolution (e.g., 640×480), and the resizer 334b ​​may be configured to resize the decoded video stream at a second resolution (e.g., 320×240). The encoder 336a is configured to encode the video stream resized by the resizer 334a to output the first output video stream 220a, and the encoder 336b is configured to encode the video stream resized by the resizer 334b ​​to output the second output video stream 220b. For example, the first output video stream 220a may be transmitted to the second client device 100′ via the second server 200′, and the second output video stream 220b may be transmitted to the third client device 100″ via the third server 200″.

[0063] With reference to FIG. 2B, the video processing server 300 is shown to include a visual effects applier 338. The visual effects applier 338 is also included in the embodiment of FIG. 2A, but is omitted for clarity. In the embodiment of FIG. 2A, the visual effects applier 338 may be bypassed or may not perform any operation on the video stream, since the visual effects have already been applied by the first client device 100 and the video stream is simply passed to the resizers 334a, 334b. In the embodiment of FIG. 2B, the video processing server 300 may be configured to process an input video stream 230 received from the first client device 100, where the input video stream 230 does not have visual effects applied on the client side (i.e., by the first client device 100), and instead the visual effects applier 338 applies one or more visual effects to the input video stream 230 (i.e., on the cloud side). In the embodiment of FIG. 2B, the video processing server 300 processes the input video stream 230 by a decoder 332 that decodes the input video stream 230. The visual effects applier 338 is configured to apply one or more visual effects to the decoded video stream according to visual effects information 230' also received from the first client device 100. The visual effects information 230' may include information regarding one or more settings of the visual effects to apply to the input video stream 230. For example, the one or more settings may include a blur radius of a background blur effect to apply to at least a portion of the input video stream 230. For example, the one or more settings may include a brightness level of a lighting effect to apply to at least a portion of the input video stream 230. Similar to FIG. 2A, the video processing server 300 is configured to resize the decoded video stream (to which the visual effects have been applied by the visual effects applier 338) by one or more resizers. In the embodiment of FIG. 2B, three resizers 334a, 334b, 334c are shown. The resizers 334 a , 334 b , 334 c are configured to perform resizing operations to obtain a desired resolution of the input video stream 230 .The desired resolution may be based on the capabilities of a client device receiving the video stream from the video processing server 300, or may be based on existing bitrate and / or bandwidth constraints for the client device receiving the video stream from the video processing server 300. For example, resizer 334a may be configured to resize the decoded video stream at a first resolution (e.g., 640×480), resizer 334b ​​may be configured to resize the decoded video stream at a second resolution (e.g., 320×240), and resizer 334c may be configured to resize the decoded video stream at an HD resolution (e.g., 1920×1080). The encoder 336a is configured to encode the video stream resized by the resizer 334a to output the first output video stream 240a, the encoder 336b is configured to encode the video stream resized by the resizer 334b ​​to output the second output video stream 240b, and the encoder 336c is configured to encode the video stream resized by the resizer 334c to output the third output video stream 240c. For example, the first output video stream 240a may be transmitted to the second client device 100' via the second server 200', the second output video stream 240b may be transmitted to the third client device 100'' via the third server 200'', and the third output video stream 240c may be transmitted to the first client device 100 via the first server 200. However, this disclosure is not limited to these example resolution values, and in some implementations, the resolution of the video stream received from first client device 100 may not be changed or resized. Also, this disclosure is not limited to the example client devices described as receiving the output video stream, and the output video stream may be sent to client devices participating in the videoconferencing session other than the example client devices described.

[0064] 3, an example block diagram of a client device and a video processing server, in accordance with one or more example embodiments of the present disclosure, is now described. In FIG. 3, a first client device 100 is shown, however, features of the first client device 100 described herein are also applicable to the second client device 100′ and the third client device 100″.

[0065] The first client device 100 may include one or more processors 110, one or more memory devices 120, a video conferencing application 130, a camera 140, an input device 150, and a display 160. The video processing server 300 may include one or more processors 310, one or more memory devices 320, and a video conferencing service provider 330.

[0066] For example, the one or more processors 110, 310 may be any suitable processing device that may be included in the first client device 100 or the video processing server 300. For example, such processors 110, 310 may include one or more of a processor, processor core, controller and arithmetic logic unit, central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), image processor, microcomputer, field programmable array, programmable logic unit, application specific integrated circuit (ASIC), microprocessor, microcontroller, etc., and combinations thereof, including any other device capable of responding to and executing instructions in a defined manner. The one or more processors 110, 310 may be a single processor, or multiple processors operatively connected, for example in parallel.

[0067] The memory 120, 320 may include one or more non-transitory computer storage media, such as read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), and volatile memory devices such as flash memory, USB drives, random access memory (RAM), hard disks, floppy disks, Blu-ray disks, or optical media such as CD ROM disks and DVDs, and combinations thereof. However, the embodiments of the memory 120, 320 are not limited to the above description, and as will be appreciated by those skilled in the art, the memory 120, 320 may be realized by a variety of other devices and structures.

[0068] For example, memory 120 may store instructions that, when executed, cause one or more processors 110 to execute a video conferencing application 130 with another client device via the video processing server 300, as described by embodiments of the present disclosure. For example, memory 320 may store instructions that, when executed, cause one or more processors 310 to provide video conferencing services to multiple client devices, as described by embodiments of the present disclosure.

[0069] Memory 120 may also include data 122 and instructions 124, which may be obtained, manipulated, created, or stored by one or more processor(s) 110. In some exemplary embodiments, such data may be accessed and used as input to execute video conferencing application 130 with another client device via video processing server 300, as described by examples of the present disclosure. Memory 320 may also include data 322 and instructions 324, which may be obtained, manipulated, created, or stored by one or more processor(s) 310. In some exemplary embodiments, such data may be accessed and used as input to provide video conferencing services to multiple client devices, as described by examples of the present disclosure.

[0070] The first client device 100 includes a videoconferencing application 130, which may also be referred to as a video telephony application or video call application. The videoconferencing application 130 allows a user of the first client device 100 to communicate with a user of another client device via transmission of audio and visual signals, thereby allowing two or more users to participate in a real-time conversation from different locations and allowing two or more users to view each other via video streams at different locations. According to an embodiment of the present disclosure, the videoconferencing application 130 may include a self-view sizer 132 and a visual effects mode selector 134. Features of the self-view sizer 132 and the visual effects mode selector 134 are described in more detail below.

[0071] The first client device 100 includes a camera 140, and the video conferencing application 130 may capture image data from the camera 140 and generate a video stream that is transmitted to the video processing server 300. In some exemplary embodiments, the camera 140 collects image data from one or more users of the first client device 100, such as one or more users' self-views. The camera 140 may be any device capable of capturing visual data. The first client device 100 is configured to generate a video stream and transmit the video stream to the video processing server 300 (e.g., via the network 400 and one or more intermediate servers, such as the first server 200). For example, the camera 140 may be an integrated webcam of the first client device 100, or a communicatively connected camera device, etc. The first client device 100 may encode the captured video (e.g., as specified by the instructions 124, etc.). In some implementations, the first client device 100 may encode the captured video in high resolution and quality (e.g., in HD format).

[0072] The first client device 100 includes an input device 150 configured to receive input from a user, and may include, for example, one or more of a keyboard (e.g., a physical keyboard, a virtual keyboard, etc.), a mouse, a joystick, a button, a switch, an electronic pen or stylus, a gesture recognition sensor (e.g., recognizing a user's gestures including movements of body parts), an input sound device or a voice recognition sensor (e.g., a microphone receiving voice commands), a trackball, a remote controller, a mobile phone (e.g., a cellular phone or a smartphone), a tablet PC, a pedal or foot switch, and a virtual reality device, etc. The input device 150 may further include a haptic device that provides haptic feedback to the user. The input device 150 may also be embodied, for example, by a touch-sensitive display having touch screen capabilities. The input device 150 may be used by a user of the first client device 100 to select one or more visual effects to apply to the video stream generated by the first client device 100. For example, the input device 150 may be used by a user of the first client device 100 to select one or more visual effects to apply to the user's self-view, such as a light and dark effect, a background blur effect, and a background replacement effect. The input device 150 may be used by a user of the first client device 100 to select default settings for when to apply visual effects on the client side and when to apply visual effects on the cloud side. For example, a user may specify default settings so that the video processing server 300 applies visual effects to a video conferencing session when the video conferencing application 130 is first run.

[0073] The first client device 100 includes a display 160 that displays information viewable by a user. For example, the display 160 may be a non-touch-sensitive display or a touch-sensitive display. The display 160 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, an active matrix organic light-emitting diode (AMOLED), a flexible display, a 3D display, a plasma display panel (PDP), and a cathode ray tube (CRT) display. However, the present disclosure is not limited to these exemplary displays and may include other types of displays.

[0074] According to example embodiments described herein, the video processing server 300 may include one or more processor(s) 310 and memory 320 as described above. The video processing server 300 may also include a video conferencing service provider 330. For example, the video conferencing service provider 330 may include a transcoder 331 and a visual effects applier 338. The transcoder 331 may be configured to receive compressed media (e.g., video) from the first client device 100, decompress the media, and apply visual effects to the uncompressed media. To implement visual effects on the video stream, the transcoder 331 may utilize a library that supports combining visual rendering effects (e.g., blurring, scaling, lighting, etc.) with performing foreground segmentation (e.g., segmenting a user and applying visual effects to the background in the user's self-view of the first client device 100). For example, the transcoder 331 may include one or more decoders 332 that decode the video streams received from the first client device 100, e.g., as discussed with respect to the embodiment of Figures 2A-2B. For example, the transcoder 331 may include one or more resizers 334 that resize the video streams received from the first client device 100, e.g., as discussed with respect to the embodiment of Figures 2A-2B. For example, the transcoder 331 may include one or more encoders 336 that encode each video stream resized by a corresponding resizer 334, e.g., as discussed with respect to the embodiment of Figures 2A-2B, which are then output to each client device participating in the video conferencing session.

[0075] The visual effects applier 338 is configured to apply one or more visual effects to the video stream received from the first client device 100 based on the visual effects information received from the first client device 100. For example, the visual effects information may be received in the form of configuration properties (e.g., via metadata) associated with the video stream. The configuration properties may include one or more settings of one or more visual effects to be applied to the video stream by the video processing server 300. When the first client device 100 requests the video processing server 300 to apply a visual effect to the video stream, the decoder 332 decodes the video stream received from the first client device 100 and passes the decoded video stream to the visual effects applier 338 (also referred to as a media pipe effects module), which applies the visual effects to the video stream to generate a modified video stream.

[0076] The video processing server 300 may be a specially configured server or servers dedicated to performing video processing on video streams received from client devices participating in a video conferencing session to apply visual effects to the video streams. For example, the video processing server 300 may include one or more CPUs, multiple GPUs (e.g., for performing segmentation operations and / or applying visual effects to the video streams), and multiple video encoder accelerators (e.g., ASICs for performing transcoding operations such as encoding, decoding, and resizing).

[0077] Further aspects of the first client device 100 and the video processing server 300 are discussed below with reference to the examples shown in FIGS. 4-5 and the flow diagrams of FIGS. 6-10.

[0078] FIG. 4 illustrates an exemplary display of a video conference session at a client device, according to one or more exemplary embodiments of the present disclosure. Referring to FIG. 4, during a video conference session 410, while executing a video conference application 130 at a first client device 100, a display 160 may display a presentation 420, a self-view 430 of a user of the first client device 100, and one or more external live stream(s) 440 of one or more other users of other client devices participating in the video conference session 410. For example, the first client device 100 may receive one or more video streams from the video processing server 300, the first server 200, or another server not specifically shown in the drawing. The respective positions and / or sizes of the presentation 420, the self-view 430, and / or the external live stream(s) 440 may be changed, for example, according to input received from a user of the first client device 100. For example, the user of the first client device 100 may change the size of the self-view 430 via the input device 150. Additionally, one or more of the presentation 420, the self-view 430, and the one or more external live stream(s) 440 may not be displayed on the display 160 of the first client device 100. Also, although FIGURE 4 shows the user of the first client device 100 as the presenter of the presentation 420, the user of the first client device 100 may be any participant in the videoconferencing session 410 and may not be the presenter.

[0079] FIG. 5 illustrates an exemplary display of a visual effect applied to a self-view of a user during a video conference session at a client device, according to one or more exemplary embodiments of the present disclosure. With reference to FIG. 5, during a video conference session, while running a video conference application 130 at a first client device 100, a display 160 may display a self-view 510 of a user of the first client device 100, the self-view 510 including a user 520 and a blurred background 530. Thus, in the exemplary self-view 510 of FIG. 5, a background blurring visual effect is applied to the background of the user 520, which may have been selected or requested by the user 520 for privacy reasons, for example. The visual effect may be applied to the background by the client device 100 or the video processing server 300 according to the operating mode of the first client device 100, according to the size of the user's self-view, and according to various other factors described below. The present disclosure is not limited to the exemplary embodiment of FIG. 5. For example, visual effects other than or in addition to a blur effect may be applied to the video stream. For example, lighting effects (eg, increasing or decreasing brightness and / or contrast values), substitute backgrounds, and other visual effects may be applied to the video stream.

[0080] 6-10 illustrate flow diagrams of exemplary, non-limiting computer-implemented methods in accordance with one or more exemplary embodiments of the present disclosure.

[0081] Referring to FIG. 6, method 600 includes operation 610 in which a server system (e.g., video processing server 300) receives a video stream regarding a video conference session from a first client device 100. The method further includes operation 620 in which the server system (e.g., video processing server 300) receives visual effect information regarding one or more visual effects to be applied to the video stream from the first client device 100. Operation 630 includes the server system (e.g., video processing server 300) applying one or more visual effects to the video stream based on the received visual effect information to generate one or more modified video streams (e.g., modified video streams 192, 194). Operation 640 includes the server system (e.g., video processing server 300) sending the one or more modified video streams to one or more other client devices (e.g., second client device 100' and / or third client device 100'') participating in the video conference session. Although not shown in FIG. 6, an optional operation may include the server system (e.g., video processing server 300) applying one or more visual effects to the video stream based on the received visual effect information to generate a modified video stream (e.g., a looped-back or modified video stream 196) and sending the modified video stream to the first client device 100.

[0082] With respect to the modified video stream 196 (i.e., loop-back video stream), the first client device 100 may be configured to receive the modified video stream 196 from the video processing server 300 by specifically requesting that the video processing server 300 send the modified video stream 196 back to the first client device 100, or the video processing server 300 may be configured to automatically send the modified video stream 196 to the first client device 100. For example, when a visual effect is requested by the first client device 100, the video processing server 300 is configured to signal (e.g., via a message or via metadata) to an intermediate server (e.g., the first server 200) that the video stream received from the first client device 100 is to be used, at least in part, to also be sent from the video processing server 300 to the client device 100 via the intermediate server. Thus, in response to receiving the signal from the video processing server 300, the intermediate server prepares to send at least a portion of the video stream back to the first client device 100, and a push notification is sent to the first client device 100 informing the first client device 100 about the new stream. When the first client device 100 receives the notification about the modified stream, the first client device 100 is configured to perform a search of the input data from the intermediate server, including the modified video stream 196. The first client device 100 may be configured to display the modified video stream 196 in preference to the video stream 190 generated locally at the first client device 100.

[0083] In an exemplary embodiment, if the first client device 100 requests the video processing server 300 to apply one or more visual effects to the video stream 190 and the video processing server 300 is unable to modify the video stream 190 (e.g., due to lack of resources or processing limitations), the video processing server 300 may be configured to forward the video stream 190 without applying the visual effects to other client devices in the video conferencing session (e.g., the second client device 100′ and / or the third client device 100″). However, in an exemplary embodiment, if the video processing server 300 is unable to modify the video stream 190 (e.g., due to lack of resources or processing limitations), the video processing server 300 may also be configured to not forward the video stream 190 to other client devices in the video conferencing session (e.g., the second client device 100′ and / or the third client device 100″). If the video processing server 300 is unable to modify the video stream 190 as requested, the video processing server 300 may receive instructions from the first client device 100 not to forward the video stream 190 to other client devices in the video conferencing session (e.g., the second client device 100′ and / or the third client device 100″). This aspect provides additional privacy to the user of the first client device 100 if the requested visual effect or effects are privacy related (e.g., the user does not want other users in the video conferencing session to see the unblurred background). Thus, other (remote) client devices in the video conferencing session may receive a modified video stream (with one or more visual effects applied by the video processing server 300 as requested by the first client device 100) or may not receive any video at all from the first client device 100 during intermittent failures of the video processing server 300.

[0084] 7, method 700 includes an operation 710 of determining whether a computing device (e.g., a computing device of a first client device 100) processes video for a video conferencing session in a first mode or a second mode. For example, the first client device 100 may be configured to toggle or switch between a first mode in which the first client device 100 requests that the video processing server 300 apply visual effects to a video stream 190 generated by the first client device 100, and a second mode in which the first client device 100 applies visual effects to the video stream itself and sends the modified video stream (e.g., the input video stream 210 of FIG. 2A) to the video processing server 300 (e.g., via the first server 200), which then resizes the modified video stream as necessary and outputs the resized modified video stream(s) to one or more other client devices participating in the video conferencing session. In some cases, the first client device 100 may send the modified video stream to one or more other client devices participating in the video conferencing session (e.g., via one or more servers, such as any of the first server 200, the second server 200', or the third server 200'') without the modified video stream passing through the video processing server. In some cases, a remote client device requesting a full resolution video stream may receive the modified video stream from the first client device 100 via another server (e.g., any of the first server 200, the second server 200', or the third server 200''), and the modified video stream does not pass through the video processing server 300 (because no resizing operation needs to be performed by the video processing server 300).

[0085] In response to determining to process the video in the first mode, at operation 720, the method includes the computing device (e.g., first client device 100) generating and sending a first video stream (e.g., input video stream 230) to a server (e.g., video processing server 300), and at operation 730, the method includes the computing device (e.g., first client device 100) sending to the server (e.g., video processing server 300) visual effects information (e.g., visual effects information 230′) regarding one or more visual effects to be applied by the video processing server 300 to the first video stream (e.g., input video stream 230), where the first video stream (e.g., input video stream 230) corresponds to a raw video stream generated with image data captured by camera 140, which may correspond to a user's self-view. Thus, in the first mode, the first client device 100 sends a raw video stream without visual effects to the video processing server 300, and further sends visual effects information to the video processing server 300 regarding one or more visual effects to apply to the first video stream.

[0086] In response to determining to process the video in the second mode, at operation 780, the method includes the computing device (e.g., first client device 100) generating a second video stream (e.g., input video stream 210) by the computing device (e.g., first client device 100) applying one or more visual effects to the first video stream (before the first video stream is actually transmitted). At operation 790, the method includes the computing device (e.g., first client device 100) transmitting the second video stream (e.g., input video stream 230) to a server (e.g., video processing server 300) or transmitting the second video stream to another server (e.g., a separate server system that may include any of the first server 200, second server 200′, third server 200″, etc.), thus bypassing the video processing server 300 entirely.

[0087] In some implementations, a computing device (e.g., the first client device 100) may be configured to operate in a mixed mode. For example, the first client device 100 may operate in a mixed mode when it switches from a first mode to a second mode in which the first client device 100 applies visual effects to the video stream. If it is determined at operation 740 that the first client device 100 operates in the mixed mode (e.g., before switching to the second mode), at operation 760, the one or more processors 110 of the first client device 100 are configured to generate a second video stream by the first client device 100 applying one or more visual effects to the first video stream, and at operation 770, the first client device 100 displays the second video stream as a self-view. At the same time, the first client device 100 transmits the first video stream and visual effect information to a server (e.g., the video processing server 300). That is, while the first client device 100 is in the mixed mode, the unmodified video stream and visual effect information continue to be sent to the video processing server 300, which in turn continues to apply one or more visual effects to the unmodified video stream, and one or more modified video streams (with the visual effects applied by the video processing server 300) are sent to be displayed at the remote client devices (e.g., the second client device 100' and / or the third client device 100'') participating in the video conference session. When the mixed mode is implemented, the switch from the first mode to the second mode is completely transparent to the remote client devices, but the self-view at the first client device 100 may experience minimal interruptions or glitches when switching to the mixed mode due to the different timelines of the video streams. In an embodiment, to avoid operating in the mixed mode for an extended period of time, the first client device 100 may be configured to switch from the mixed mode to the second mode after a certain predetermined time (e.g., after 5 minutes).When the self-view is determined to be a size below the threshold, this approach prevents the waste of cloud resources and thus allows the first client device 100 to apply visual effects without degrading the self-view too much.

[0088] If, at operation 740, it is determined that the first client device 100 is not operating in mixed mode, then, at operation 750, the first client device 100 receives a modified video stream (e.g., modified video stream 196) from a server (e.g., video processing server 300) and displays the modified video stream as a self-view on the display 160. For example, the modified video stream includes one or more visual effects applied to the first video stream by the server (e.g., video processing server 300).

[0089] 8, the method 800 includes an operation 810 in which a computing device (e.g., a computing device of the first client device 100) determines whether a size of the self-view exceeds a threshold level. For example, the visual effects mode selector 134 may be configured to toggle or switch between a first mode in which the first client device 100 requests that the video processing server 300 apply visual effects to a video stream generated by the first client device 100 based on the size of the self-view and / or the physical resolution of the display of the first client device 100, and a second mode in which the first client device 100 applies visual effects to the video stream and sends the modified video stream to the video processing server 300. The size of the self-view may correspond to an active resolution of the self-view and may be determined by the self-view size determiner 132. For example, during a video conference session, when the size of the self-view displayed on the display of the client device exceeds a threshold level (value), the operation 820 of the computing device (e.g., the computing device of the first client device 100) includes the visual effects mode selector 134 determining to switch to the second mode (or to maintain the second mode). When the size of the self-view exceeds the threshold level (value), a video quality problem perceptible by a user may occur at the client device. Conversely, when the size of the self-view displayed on the display of the first client device 100 is less than the threshold level (value), the operation 830 of the computing device (e.g., the computing device of the first client device 100) includes the visual effects mode selector 134 determining to switch to the first mode (or to maintain the first mode).

[0090] In some aspects, in response to the media quality of the modified video stream (e.g., self-view) being below a threshold level and / or the latency of the modified video stream received from the video processing server 300 exceeding a threshold level, the one or more processors 110 are configured to determine to switch to the second mode and / or stop sending the video stream to the video processing server 300. For example, whether the quality of the modified video stream (e.g., self-view) is below a threshold may be determined according to various factors and metrics related to quality of service and quality of experience (e.g., whether the resolution of the self-view is below a threshold level, whether the self-view freezes a certain number of times within a predetermined time, whether the self-view freezes for longer than a predetermined time, the signal-to-noise ratio of the self-view, etc.).

[0091] In some implementations, in response to the media quality of the modified video stream sent to the first client device 100 being below a threshold level and / or the latency of the modified video stream sent to the first client device 100 exceeding a threshold level, the video processing server 300 is configured to send a notification to the first client device 100 indicating that the first client device 100 will assume control for applying one or more visual effects to the video stream (switching to the second mode) and / or that the video processing server 300 will stop transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session (e.g., the second client device 100' and the third client device 100'').

[0092] In an embodiment, when multiple self-views are displayed on the display of the first client device 100 during a video conference session, the self-view sizer 132 is configured to identify a self-view with a maximum size. The one or more processors 110 of the first client device 100 are configured to determine whether the size of the self-view with the maximum size exceeds a threshold level (value) to determine whether to switch to (or remain in) the second mode. The self-view sizer 132 may be configured to track or monitor the size of one or more self-views during the video conference session to determine whether there is a change in the maximum size and whether to switch to (or remain in) the second mode. Changes regarding the size and / or position of the self-view may be transmitted to the video processing server 300 to update the position of the visual effects applied to the self-view when the first client device is in the first mode. In an embodiment, when the size of the self-view is frequently changed by the user (i.e., changed more than a predetermined number of times within a predetermined time, such as more than three times within 10 seconds), the self-view sizer 132 may be configured to wait until the resizing has stopped for a predetermined time (e.g., 3 seconds) before determining the size of the self-view. The one or more processors 110 may then determine whether the size of the self-view displayed on the display of the first client device 100 exceeds a threshold level.

[0093] Referring to FIG. 9, the method 900 includes an operation 910 in which a computing device (e.g., a computing device of the first client device 100) determines whether a zoom function (e.g., an auto-zoom or centering function) is enabled in a video conferencing application at the first client device 100. That is, when the one or more processors 110 determine that the zoom function is enabled, an operation 920 may include the visual effects mode selector 134 switching to a second mode or maintaining the second mode. For example, the video processing server 300 may not be allowed to apply visual effects to the enlarged video stream in the first mode because applying visual effects and resizing the enlarged video stream may result in adverse effects of quality reduction (e.g., resolution reduction) in the modified video stream. When the one or more processors 110 determine that the zoom function is not enabled, an operation 930 may include the visual effects mode selector 134 allowing the first client device 100 to switch to the first mode or maintaining the first mode.

[0094] In an alternative embodiment, when a zoom function is enabled in the video conferencing application at the first client device 100, the video processing server 300 may be configured to apply visual effects to the video stream in the first mode. For example, a magnified stream (i.e., a raw video stream modified with zoom at the first client device 100) may be sent from the first client device 100 to the video processing server 300, and visual effects may be applied to the magnified stream. For example, a raw video stream may be sent from the first client device 100 to the video processing server 300, and the video processing server 300 may apply both the magnification and visual effects to the video stream.

[0095] 10, the method 1000 includes an operation 1010 of a computing device (e.g., a computing device of the first client device 100) in which one or more processors 110 determine whether a user of the first client device 100 is in a waiting room or participating in a video conference session. For example, when switching between a first mode and a second mode, the first client device 100 may be configured to distinguish between a "waiting room" or "waiting room" state in which the user is waiting to be admitted or join a meeting of the video conference session and a "joining" state in which the user has been admitted or is participating in the meeting. For example, in some implementations, when the first client device 100 is in a waiting state, the video processing server 300 may not support sending or receiving media to or from the first client device 100 while the first client device 100 remains in the waiting state. For example, in response to the one or more processors 110 determining that the first client device 100 has transitioned from a waiting room to a joining state, the visual effects mode selector 134 may be configured to switch from the second mode to the first mode or to maintain the first mode at operation 1020. For example, in response to the one or more processors 110 determining that the first client device 100 is in a waiting room state, the visual effects mode selector 134 may be configured to switch from the first mode to the second mode or to maintain the second mode at operation 1030. For example, a background blur effect (or other visual effect) applied to a video stream by the first client device 100 while the first client device 100 was waiting in the waiting room to be admitted into a meeting "transitions" from the first client device 100 to the video processing server 300 in response to the first client device 100 being admitted into a meeting of a video conferencing session. Thus, when a first client device 100 joins a meeting, the video processing server 300 is responsible for applying a background blur effect (or other visual effect) to the video stream.In an alternative embodiment, the video processing server 300 may support sending and receiving media to the first client device 100 while the first client device 100 is in a standby state, and thus the operations of FIG. 10 may not be performed.

[0096] When the first client device 100 toggles or switches between the first and second modes, intermittent video problems (e.g., video freezing, frame drops, etc.) may occur. Such intermittent video problems may be more apparent in the self-view than in the view of a remote participant of the video conference session. The exemplary embodiments described below provide additional ways to avoid or mitigate these video problems that occur when switching between the first and second modes. For example, when switching from the first mode to the second mode (or vice versa), the first client device 100 may be configured to wait until the second mode (or the first mode) begins and then stop the previous implementation. For example, when switching from the second mode to the first mode, the first client device 100 begins receiving remote frames of the self-view generated by the video processing server 300 before the first client device 100 stops applying visual effects to the video stream that it sends to the video processing server 300. This may result in some frames of the video stream(s) received by the first client device 100 and the remote client devices (e.g., the second client device 100′ and the third client device 100″) having a double visual effect applied to the self-view of the user of the first client device 100. For example, when switching from the first mode to the second mode, the first client device 100 starts generating and displaying frames of the self-view with the visual effect applied to the video stream by the first client device 100 before the video processing server 300 stops applying the visual effect to the video stream sent to the first client device 100 and looped back to the first client device 100. This may also result in some frames of the video stream(s) received by the first client device 100 and the remote client devices having a double visual effect applied to the self-view of the user of the first client device 100.

[0097] For example, if the first client device 100 is switched by the video processing server 300 to a first mode of applying background blur to the video stream while the first client device 100 is applying a background replacement visual effect to the video stream, the visual effects mode selector 134 of the first client device 100 may be configured to temporarily operate in a hybrid or mixed mode, which allows the video processing server 300 to apply background blur to the video stream and at the same time, the first client device 100 still applies the background replacement visual effect to the video stream for a predetermined time (or until a first key frame of the remote modified video stream generated by the video processing server 300 applying the background blur to the video stream is received by the first client device 100). After expiration of the predetermined time (or after the first key frame is received), the visual effects mode selector 134 is configured to disable the background replacement visual effect applied by the first client device 100, so that the first client device 100 operates in the first mode. Thus, some frames of the video stream may briefly contain both visual effects, but such an approach ensures that unprocessed frames are not forwarded to other remote client devices participating in the video conferencing session (e.g., the second client device 100' and the third client device 100'').

[0098] In an exemplary embodiment, when switching from the second mode to the first mode, the one or more processors 110 of the first client device 100 may be configured to continue displaying the self-view of the user of the first client device 100 via the local stream until the first keyframe of the remote modified video stream is received from the video processing server 300. This approach results in no freezing and can minimize the number of frames with double effects applied (e.g., double blur frames with visual effects applied by both the first client device 100 and the video processing server 300). If the video stream has not yet been resized for multiple resolutions, the remote views at other client devices (e.g., the second client device 100' and the third client device 100'') in the video conference session may temporarily freeze when the first client device 100 requests that the video processing server 300 apply a visual effect, and may recover from the freeze when the first keyframe is propagated from the first client device 100 to the other client devices via the video processing server 300. The first client device 100 continues to apply the visual effect until it receives a new modified video stream from the video processing server 300. Thus, the first client device 100 receives some frames with the effect applied twice. In another embodiment, the view request for the self-view can be limited to a threshold resolution (e.g., 640×360) to make the switching from the second mode to the first mode in the self-view faster.

[0099] For example, when switching from the first mode to the second mode, the one or more processors 110 of the first client device 100 may be configured to display a self-view using the local stream as soon as the first client device 100 applies the visual effects, without any artifacts being perceptible. The remote views at other client devices in the video conference session (e.g., the second client device 100' and the third client device 100'') may display some frames with double effects applied (e.g., double blur) and some stalls / frame drops while the first client device 100 loads the visual effects, until the disablement of the visual effects by the video processing server 300 is performed.

[0100] In some implementations, other configurations in the first client device 100 and / or the video processing server 300 may be implemented to reduce the freeze time and / or reduce or avoid double effect frames. For example, the video processing server 300 may be kept in a state ready to apply visual effects (e.g., ready state) to the video stream transmitted from the first client device 100, but does not actually apply any visual effects until the mode is switched to the first mode. Keeping the video processing server 300 in a state ready to apply visual effects reduces the startup time of the video processing server 300 and reduces the freeze time of the remote client devices (e.g., the second client device 100' and the third client device 100'') during the video conference session. In an embodiment, the first client device 100 may be configured to send a timestamp / frame counter to the video processing server 300 indicating a future time at which the first client device 100 would like to switch modes. The video processing server 300 may be configured to ensure that visual effects are enabled / disabled in that frame at the appropriate time. The first client device 100 may be configured to wait until the timestamp / frame counter value before starting / stopping the application of the visual effect, thus preventing double effect frames. According to this embodiment in which the first client device 100 sends the timestamp / frame counter, the video processing server 300 may be configured to switch modes immediately and / or may remain in a ready state.

[0101] In an embodiment, the first client device 100 may be configured to send metadata for frames of the video stream, which provides information about the visual effect to the video processing server 300. When the first client device 100 requests the video processing server 300 to apply a visual effect, the video processing server 300 is configured to generate a modified video stream with the visual effect only if the applied visual effect is different from any visual effect applied to the input video stream. According to this embodiment, in which the first client device 100 sends metadata for frames of the video stream that provides information about the visual effect to the video processing server 300, the video processing server 300 may be configured to switch modes instantly and / or remain in a ready state.

[0102] For example, in some implementations, the first client device 100 and / or the video processing server 300 may preload visual effects without actually applying the visual effects to the output stream. For example, when switching from the first mode to the second mode, the first client device 100 may be configured to preload visual effects, request the video processing server 300 to freeze on the current frame on which the visual effects have been applied by the video processing server 300, transmit the video stream on which the visual effects have been applied by the first client device 100, and then request the video processing server 300 to stop applying the visual effects to the video stream. Thus, instead of a double effect frame, a short video freeze is added. For example, when switching from the second mode to the first mode, the video processing server 300 may be configured to receive a request from the first client device 100 to preload visual effects and freeze the frame on which the visual effects have been applied by the first client device 100. The first client device 100 may stop applying visual effects to the video stream and send the video stream without the visual effects to the video processing server 300. The video processing server 300 is then asked to apply visual effects to the video stream, thus adding a short video freeze instead of a double effect frame.

[0103] In some embodiments, the video conferencing application 130 may have default settings for the first client device 100 to operate in the first mode. That is, the video conferencing application 130 may preferably, by default, have the video processing server 300 apply visual effects in the first mode during a video conferencing session (rather than the first client device 100 applying visual effects to the video stream in the second mode). In response to the video processing server 300 being unavailable (e.g., due to resource shortage, high peak usage, bandwidth limitation, etc.), the visual effect mode selector 134 may be configured to switch to the second mode, whereby the processing of visual effects is applied by the first client device 100. For example, in response to a shortage of graphic computing resources in the video processing server 300, the video processing server 300 notifies the first client device 100 (e.g., via the first server 200) of the unavailability of the first mode by means of a message and metadata, etc., and the visual effect mode selector 134 switches to the second mode in which the first client device 100 applies visual effects (i.e., client-side effects). In an exemplary embodiment, if the first client device 100 cannot support the visual effects required to be applied to the video stream, one or more processors 110 of the first client device 100 may be configured to control the display 160 of the first client device 100 to display a message indicating that the required visual effects cannot be applied to the video stream, and the camera 140 of the first client device 100 may be muted to protect the user's privacy. Additionally or alternatively, if the first client device 100 cannot support the visual effects required to be applied to the video stream, one or more processors 110 of the first client device 100 may be configured to stop transmitting the video stream. The first client device 100 may notify the user in an additional or alternative manner (e.g., via a speaker) that the required visual effects cannot be applied to the video stream.

[0104] In some embodiments, the first client device 100 may be configured to display a self-view of the user of the first client device 100 during a video conferencing session. The self-view may be generated and displayed locally by the first client device 100, or the self-view may correspond to a video stream modified by the video processing server 300 applying one or more visual effects to the video stream, and is displayed as the user's self-view by looping back the modified video stream to the first client device 100. For example, when the first client device 100 operates in a first mode and requests that the video processing server 300 apply a visual effect to the video stream, the first client device 100 may be configured to receive the modified video stream "remotely". The modified video stream received remotely may have degraded quality and latency compared to the local self-view generated by the first client device 100 due to network round-trip and transcoding performed by the video processing server 300. However, according to the exemplary embodiments disclosed herein, the quality and latency of the self-view can be improved according to various techniques. Thus, the media quality and latency of the self-view can be maintained at a level that does not adversely affect the user's perception of the quality of the video conferencing session. For example, to obtain a lower latency (e.g., a threshold of less than 1 second, a threshold of less than 300 ms, etc.), the video processing server 300 may be configured to enable WebRTC low latency rendering and set a latency threshold limit when sending the modified video stream back to the first client device 100 (e.g., by setting the threshold limit with the PlayoutDelayLimits RTP header extension).As an additional or alternative approach to improving self-view quality, the video processing server 300 may be configured to adjust its bitrate allocation strategy to prioritize sending the modified video stream back to the first client device 100 over sending the modified video stream to other client devices (e.g., the second client device 100′ and the third client device 100″). That is, the video processing server 300 may be configured to prioritize the modified video stream 196 that is looped back when distributing available bandwidth downstream. Without this prioritization, in larger meetings, the user of the first client device 100 loses self-view when not speaking, but other users in the video conferencing session can still see the user. As an additional or alternative approach to improving self-view quality, one or more processors 110 of the first client device 100 may be configured to reduce the frame rate of remotely received video streams (e.g., video streams transmitted from other client devices in the video conference session, such as the second client device 100' and the third client device 100'') as a performance adaptation, but excluding the modified video stream of the self-view (i.e., maintained at a default frame rate).If, despite one or more of the above-mentioned techniques being implemented (e.g., by increasing the priority of the modified video stream to send back to the first client device 100), there is still not enough bandwidth for the modified video stream 196 to be looped back, the one or more processors 110 of the first client device 100 may be configured to detect that the video processing server 300 and / or the first server 200 are unable to modify the video stream 190 generated by the first client device 100 with visual effects, the visual effects mode selector 134 may switch from processing the video in a first mode to processing the video in a second mode, and the first client device 100 may modify the video stream by applying visual effects to the video stream to generate its own self-view.

[0105] Terms such as "module" and "unit" may be used herein in connection with various features of the present disclosure. Such terms may refer to software or hardware components or devices that perform specific tasks, such as, but not limited to, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A module or unit may be configured to reside on an addressable storage medium and configured to run on one or more processors. Thus, a module or unit may include, by way of example, components such as software components, object-oriented software components, class and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided in the components and modules / units may be combined into fewer components and modules / units or further separated into additional components and modules.

[0106] Aspects of the exemplary embodiments described above may be recorded on a computer-readable medium (e.g., a non-transitory computer-readable medium) that includes program instructions for performing various computer-implemented operations. The medium may also include program instructions, data files, data structures, and the like, alone or in combination. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD ROM disks, Blue-Ray disks, and DVDs, magneto-optical media such as optical disks, and other hardware devices specially configured to store and execute program instructions, such as semiconductor memory, read-only memory (ROM), random access memory (RAM), flash memory, and USB memory. Examples of program instructions include both machine code, such as generated by a compiler, and files containing higher-level code that may be executed by a computer using an interpreter. The program instructions may be executed by one or more processors. To perform the operations of the embodiments described above, the hardware devices described may be configured to function as one or more software modules, or vice versa. Furthermore, the non-transitory computer-readable storage medium may be distributed among computer systems connected over a network, and the computer-readable code or program instructions may be stored and executed in a decentralized manner. Furthermore, the non-transitory computer-readable storage medium may also be embodied in at least one application specific integrated circuit (ASIC) or field programmable gate array (FPGA).

[0107] Each block in the flowchart diagram may represent a unit, module, segment, or portion of code that includes one or more executable instructions for implementing a particular logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may be performed out of order. For example, two blocks shown in succession may in fact be performed substantially in parallel (concurrently), or the blocks may sometimes be performed in the reverse order, depending on the functionality involved.

[0108] Although the present disclosure has been described with respect to various exemplary embodiments, each example is provided for the purpose of explanation and not for the purpose of limiting the present disclosure. Those skilled in the art, upon understanding the above content, can easily create modifications, variations, and equivalents of such embodiments. Therefore, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the disclosed subject matter, as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. one or more memories configured to store instructions; one or more processors configured to execute the instructions stored in the one or more memories; wherein the one or more processors execute the instructions to: receiving a video stream for the video conferencing session from a first client device; receiving visual effects information from the first client device regarding one or more visual effects to apply to the video stream; applying the one or more visual effects to the video stream based on the received visual effects information to generate one or more modified video streams; transmitting the one or more modified video streams to one or more other client devices participating in the video conferencing session; The server system.

2. The server system of claim 1 , wherein the one or more processors are configured to transmit one of the one or more modified video streams to the first client device.

3. In response to a media quality of the modified video stream sent to the first client device being below a threshold level and / or a latency of the modified video stream sent to the first client device exceeding a threshold level, the one or more processors: sending a notification to the first client device indicating that the first client device has assumed control for applying the one or more visual effects to the video stream and / or to stop transmitting the one or more modified video streams to the one or more other client devices participating in the video conferencing session; The server system according to claim 2 , configured to:

4. The one or more processors: generating a plurality of modified video streams, each having a different resolution, based on the received visual effects information; For each of the one or more other client devices participating in the video conference session, based on the data transfer speed between the server system and each client device, transmitting one of the plurality of modified video streams; The server system according to claim 1, configured to perform.

5. The server system according to claim 1, wherein the visual effect information includes information regarding one or more settings of the one or more visual effects to be applied to the video stream.

6. The server system according to claim 5, wherein the one or more settings include a blur radius of a background blur effect to be applied to at least a portion of the video stream.

7. The server system according to claim 5, wherein the one or more settings include a luminance level of a light and dark effect to be applied to at least a portion of the video stream.

8. The server system according to claim 1, wherein the video stream transmitted from the first client device is a self-view of the user of the first client device.

9. In response to the self-view of the user being changed to a size exceeding a threshold level, the one or more processors Stopping the application of the one or more visual effects to the video stream; Sending a notification to the first client device indicating that the first client device is responsible for controlling the application of the one or more visual effects to the video stream and / or stopping the transmission of the one or more modified video streams to the one or more other client devices participating in the video conference session; The server system according to claim 8, configured to perform.

10. One or more memories configured to store instructions; One or more processors configured to execute the instructions stored in the one or more memories; A computing device comprising: wherein the one or more processors execute the instructions to Determine whether to process a video related to a video conference session in a first mode or a second mode; in response to determining to process the video in the first mode, generating a first video stream, transmitting the first video stream to a server system, and transmitting visual effects information to the server system regarding one or more visual effects to apply to the first video stream; in response to determining to process the video in the second mode, generating a second video stream by applying the one or more visual effects to the first video stream and transmitting the second video stream to the server system or a separate server system; The computing device.

11. 11. The computing device of claim 10, wherein the one or more processors are configured to receive from the server system a modified video stream in which the one or more visual effects have been applied to the first video stream based on the visual effects information transmitted to the server system.

12. 12. The computing device of claim 11, wherein the one or more processors are configured to determine to process the video in the second mode and / or to stop transmitting the first video stream to the server system in response to a media quality of the modified video stream being below a threshold level and / or a latency of the modified video stream received from the server system exceeding a threshold level.

13. The computing device of claim 10 , wherein the visual effects information includes information regarding one or more settings of visual effects to apply to the first video stream.

14. The computing device of claim 13 , wherein the one or more settings include a blur radius of a background blur effect to apply to at least a portion of the first video stream.

15. The computing device of claim 13 , wherein the one or more settings include a brightness level of a lighting effect to apply to at least a portion of the first video stream.

16. the video includes a self-view of a user of the computing device; the one or more processors are configured to determine whether to process the video in the first mode or the second mode based on a size of the self-view. The computing device of claim 10.

17. when the size of the self-view is below a threshold level, the one or more processors are configured to determine to process the video in the first mode; when the size of the self-view exceeds a threshold level, the one or more processors are configured to determine to process the video in the second mode.

17. The computing device of claim 16.

18. the video includes a plurality of self-views of a user of the computing device; the one or more processors are configured to identify a self-view from among the plurality of self-views having a largest size; the one or more processors are configured to determine whether to process the video in the first mode or the second mode based on a size of the self-view having the maximum size. The computing device of claim 10.

19. 11. The computing device of claim 10, wherein the visual effects information includes instructions for the server system not to transmit the first video stream to other client devices participating in the video conferencing session if the one or more visual effects cannot be applied to the first video stream.

20. determining whether to process video for the videoconferencing session in a first mode or a second mode; in response to determining to process the video in the first mode, generating a first video stream, transmitting the first video stream to a server system, and transmitting visual effects information to the server system regarding one or more visual effects to apply to the first video stream; in response to determining to process the video in the second mode, generating a second video stream by applying the one or more visual effects to the first video stream and transmitting the second video stream to the server system or a separate server system; 4. A computer-implemented method comprising:

Images