Enhance live video image streams with visual effects

Abstract

A device, method, and non-temporary program storage device are disclosed for augmenting a live video image stream using visual effects that are directly synthesized into the video image stream. For example, a first electronic device may acquire a video image stream. The electronic device may then, for each of one or more images in the video stream, perform a segmentation operation on the image to identify a foreground and background portion of the image, assign a first depth value to the foreground portion of the image (and optionally a second depth value to the background portion), augment the image with at least a first visual effect to which a third depth value is assigned, and then synthesize at least (a) the rendering of the foreground portion of the image at the first depth value and (b) the rendering of the first visual effect at the third depth value into an augmented output image.

Description

Technical Field

[0001] The present disclosure generally relates to the field of audio and video data streaming. More particularly, and without limitation, the present invention relates to techniques for augmenting live video image streams with various visual effects, such as depth perception visual effects synthesized directly into the video image stream.

Background Art

[0002] With the advent of portable integrated computing devices, cameras and other video-capturable devices have become widespread. These integrated computing devices generally take the form of smartphones, tablets, or laptop computers and typically include a general-purpose computer, a camera, a high-performance user interface including a touch-sensitive screen, and wireless communication capabilities via Wi-Fi, Bluetooth, LTE, HSDPA, New Radio (NR), and other cellular-based or wireless technologies. The widespread adoption of these integrated devices provides opportunities to use the capabilities of the devices to perform tasks that would otherwise require dedicated hardware and software.

[0003] For example, portable integrated computing devices such as smartphones, tablets, and laptops typically have two or more built-in cameras. These cameras generally become lens / camera hardware modules that can be controlled through the use of a general-purpose computer that includes firmware and / or software (e.g., an application, or "app") and a user interface including touchless controls such as touchscreen buttons, fixed buttons, and / or voice control. The integration of high-quality cameras into these portable integrated communication devices such as smartphones, tablets, and laptop computers has enabled users to capture and share images and videos in ways that were previously impossible. Currently, it is common for a user's smartphone to be their primary image capture device of choice.

[0004] With the rising popularity of photo and video sharing via portable integrated computing devices with integrated cameras, video conferencing (and other audiovisual (AV) content sharing sessions) via such devices is also on the rise. In particular, users often engage in video conferencing calls or meetings to share video images and / or other graphical content, and the video images are typically captured by a forward-facing camera on the device, i.e., a camera facing the same direction as the display screen of the camera device. Most conventional cameras are optimized for either wide-angle general photography or narrow-angle photography, e.g., self-portraits and video conferencing streaming use cases. These cameras optimized for wide-angle are typically optimized for group and landscape compositions, but are not optimal for individual portraits due to distortion that occurs, for example, when the subject is short distance from the camera or at the edge of the camera's field of view.

[0005] Therefore, in some cases, users can benefit from greater flexibility in selecting image capture sources to use in video conferencing (or other AV content sharing) sessions, particularly from the ability to leverage higher-quality image capture devices during such sessions (e.g., a camera built into a device different from the device hosting the session, such as one of the user's portable communication devices). By doing so, users may be able to stream higher-quality audio and / or video images to a second electronic device for subsequent presentation, storage, or further transmission by the second electronic device.

[0006] However, there is a greater need for the ability to enhance live video image streams in various ways, such as enhancement using depth-aware visual effects directly synthesized into the video image stream, enhancement using real-time virtual lighting effects directly synthesized into the video image stream, improved resolution of visual content transmitted as part of the video image stream, and improved AV synchronization in enhanced video image streams that include video and / or audio content from multiple sources synthesized into a single live video image stream. [Overview of the project]

[0007] Devices, methods, and non-temporary program storage devices (NPSDs) are disclosed herein for enabling the enhancement of a live video image stream with various visual effects and / or other enhancements, such as visual effects and enhancements that can be directly synthesized into the video image stream, and as a result the enhanced video image stream can be displayed on the device of the user who captured the live video image stream and / or transmitted to another user's device for display.

[0008] For example, a first image processing method is disclosed herein, the first image processing method comprising: acquiring a video image stream containing a plurality of images of a scene captured by a first image capture device in a first electronic device; then, with respect to at least a first image of the video image stream, performing a segmentation operation on the first image to identify at least a foreground portion and a background portion of the first image; assigning a first depth value in the scene to the foreground portion of the first image; (optionally) assigning a second depth value in the scene to the background portion of the first image; extending the first image with at least a first visual effect to which a third depth value in the scene is assigned; and at least compositing renderings of (a) the foreground portion of the first image at the first depth value and (b) the first visual effect at the third depth value into a first extended output image; and transmitting the first extended output image to a second electronic device. According to some embodiments, the method may further include augmenting a first image with at least a second visual effect, the second visual effect being assigned a fourth depth value in the scene and composited with the first augmented output image at the fourth depth value, the third and fourth depth values ​​being different from each other.

[0009] According to other embodiments, the first image of the video image stream may be visually enhanced before being acquired by the first electronic device by at least one of the following methods: trimming according to one or more predetermined framing rules, applying distortion correction, or applying tone mapping.

[0010] In further embodiments, the first image capture device may be connected to the first electronic device by one of the following methods: a wired connection, a wireless connection, or by being incorporated into the first electronic device.

[0011] According to some embodiments, a first augmented output image is transmitted to a second electronic device as part of a video conferencing application. According to some such embodiments, the first visual effect may include a graphical window containing visual content, which is rendered to the first augmented output image at least partially at a second resolution independent of the first resolution at which the visual content is displayed on the first electronic device.

[0012] According to another embodiment, the first depth value is assigned based on the estimated depth in the scene of the foreground portion of the first image.

[0013] In yet another embodiment, the third depth value is at least one of the following: (1) smaller than the first depth value, (2) larger than the first depth value, (3) between the first and second depth values, or (4) equal to the first depth value.

[0014] In further embodiments, the first visual effect includes at least one of (1) a virtual lighting effect, (2) a graphical window containing visual content, or (3) a graphical representation of a human subject's reaction captured within a scene.

[0015] According to a further embodiment, the first visual effect includes a graphical window containing visual content, and the third depth value is greater than the first depth value.

[0016] According to some embodiments, the foreground portion of the first image includes at least one human subject. According to some such embodiments, the first visual effect includes a graphical window containing visual content, and the foreground portion is cropped before rendering based on the size and location of the detected face of the human subject. According to other such embodiments, the first visual effect includes a graphical representation of the human subject's reaction, and at least one of (1) a third depth value or (2) the arrangement of the graphical representation of the human subject's reaction is at least partially based on the size or location of the human subject. According to yet another such embodiment, the first visual effect includes a graphical representation of the human subject's reaction, and the graphical representation includes emojis or images. According to yet another such embodiment, the first visual effect may include a virtual lighting effect, the virtual lighting effect includes at least one of (1) a virtual light color, (2) a virtual light intensity, or (3) a virtual light arrangement, and extending the first image with the virtual lighting effect further includes estimating a set of surface normals for the human subject in the first image.

[0017] According to other embodiments, the first visual effect includes a graphical window containing visual content, the visual content of the graphical window being selected via an application programming interface (API) or operating system (OS) level function of the first electronic device.

[0018] In further embodiments, the first visual effect includes a graphical window containing first audiovisual (AV) content, the video image stream includes second AV content, and the timing of the audio component of the first AV content is adjusted at least partially based on the timing of the audio component of the second AV content before being transmitted to the second electronic device. In some implementations, the timing of the audio component of the first AV content may be adjusted specifically at least partially based on the moving average difference between (1) the timing of the audio component of the first AV content and (2) the timing of the audio component of the second AV content.

[0019] Various embodiments of non-temporary program storage devices are also disclosed herein. Such NPSDs are readable by one or more processors. Instructions can be stored in the NPSD to cause one or more processors to carry out any of the embodiments disclosed herein. Various electronic devices are also disclosed herein, for example, comprising memory, one or more processors, an image capture device, a display, and / or other electronic components, and programmed to perform according to the various methods and embodiments of the NPSD disclosed herein. [Brief explanation of the drawing]

[0020] [Figure 1] This document illustrates an exemplary electronic device configuration for establishing a secure connection between electronic devices to stream audio and / or video image data, according to one or more embodiments.

[0021] [Figure 2] This document presents various examples of extended live video image streams in a graphical window containing visual content, according to one or more embodiments.

[0022] [Figure 3A]Illustrative visual effects that can be applied to enhance a live video stream according to one or more embodiments are shown. [Figure 3B] Illustrative visual effects that can be applied to enhance a live video stream according to one or more embodiments are shown.

[0023] [Figure 4] Illustrative visual effects configured to improve the resolution of visual content transmitted as part of a live video stream according to one or more embodiments are shown.

[0024] [Figure 5] Illustrative virtual lighting effects that can be applied to enhance a live video stream according to one or more embodiments are shown.

[0025] [Figure 6A] Illustrative techniques for performing improved audio - visual (AV) synchronization in an extended live video stream according to one or more embodiments are shown. [Figure 6B] Illustrative techniques for performing improved audio - visual (AV) synchronization in an extended live video stream according to one or more embodiments are shown.

[0026] [Figure 7] A flowchart showing a method for extending a live video image stream with depth - recognition visual effects according to various embodiments.

[0027] [Figure 8] A block diagram showing a programmable electronic computing device in which one or more of the technologies disclosed herein can be implemented.

Best Mode for Carrying Out the Invention

[0028] The following description includes many specific details for illustrative purposes to enhance understanding of the invention disclosed herein. However, it will be apparent to those skilled in the art that the invention can be practiced without these specific details. Where else, structures and devices are shown in block diagram form to avoid obscuring the invention. References of numbers without subscripts or suffixes are understood to refer to all cases of corresponding subscripts and suffixes. Furthermore, the language used in this disclosure has been selected primarily for readability and explanatory purposes, and not to delineate or limit the gist of the invention, and it may be necessary to rely on the claims to determine such gist of the invention. Wherever “one embodiment” or “one embodiment” (or similar) is used herein, it means that a particular feature, structure or characteristic described in relation to an embodiment is included in at least one embodiment of the invention, and whereever “one embodiment” or “one embodiment” is used multiple times, it should not be understood that all of them refer to the same embodiment.

[0029] The techniques disclosed herein generally relate to enhancing live video image streams (e.g., as part of a video conferencing session) with specific visual effects (e.g., adding graphical overlays, performing specific image processing effects related to virtual lighting and / or camera angles, extracting body parts of human subjects from a video stream and compositing the extracted parts with other graphical content in the output enhanced live video image stream), and improving the resolution and / or AV synchronization of composite AV content transmitted as part of the enhanced live video image stream.

[0030] In some cases, video image augmentation may include segmenting specific objects (or classes of objects such as foreground and background objects) from a video image into multiple layers; assembling the multiple layers of the original video image according to a desired compositing technique or visual effect (for example, by manipulating the virtual depth of one or more of such layers in the z-axis direction of the scene); applying the desired augmentation to one or more of the layers; and then compositing the various layers into the augmented video image before transmission, so that a standard video image stream can be transmitted, for example, to another device over a network.

[0031] Exemplary device setup for establishing secure audio-visual (AV) connectivity between electronic devices

[0032] Next, referring to Figure 1, an exemplary electronic device configuration 100 for establishing a secure connection between electronic devices to stream audio and / or video image data is shown according to one or more embodiments.

[0033] Turning to the first simplified scenario 1A (100A), the first device 102 includes a built-in image capture device 106 (e.g., a “webcam”) which may be connected inside other components of the first device 102 and may be used to capture one or more exemplary images (109) of a scene surrounding the first device 102, for example, including one or more human subjects.

[0034] Next, referring to scenario 1B (100B), the first device 102 also includes an embedded image capture device 106, but also forms a secure wired connection, for example, via a wired connection 107B, with a second device 104B which itself has an embedded image capture device 105 (and may also have one or more additional embedded image capture devices).

[0035] In some cases, the image capture device 105 of the second device may be of higher quality than the image capture device 106 of the first device, for example, with respect to resolution, zoom, field of view (FOV), spatial resolution, focus, color quality, or any other imaging parameters. In such cases, it may be more desirable to use the image capture device 105 of the second device rather than the image capture device 106 of the first device to capture images that are used, extended, displayed, stored, transmitted, etc., by the first device (102) as part of an active AV communication session.

[0036] In other cases, the user of the first device 102 may simply want to select the image capture device of the second device 104B, for example, to provide a different (and / or additional) view of the scene surrounding the first device 102, for any number of other reasons, such as the second device 104B not having its own image capture device, the image capture device of the first device 102 not functioning properly, or the image capture device of the first device 102 having special image capture functions or modes desired by the user of the second device (e.g., "portrait" or shallow depth of field (SDOF) photography mode by composite, "night" capture mode, "slow motion" video capture mode, etc.).

[0037] Referring here to Scenario 1C (100C), the first device 102 and the second device 104C are in close proximity to each other and attempt to form a secure wireless connection, e.g., wireless connection 107C, via an agreed wireless connection protocol. As used herein, the term “in close proximity” can refer to devices that are discoverable from each other with respect to a given wireless connection protocol, for example, as indicated by a boundary circle 108 having a radius of 110. According to some implementations, measured signal strength can be used as a proxy to estimate the distance between two devices and determine whether they are within a sufficiently close range to each other. In some such implementations, the threshold signal strength required to determine that two devices are in sufficiently close proximity to each other may not be a fixed threshold and may also be based on filtering (e.g., averaging) of signal strength values ​​over time across many signal strength samples. In some embodiments, the first device 102 may comprise one or more image capture devices (e.g., 106), and the second device 104C may also comprise one or more image capture devices (e.g., 105).

[0038] In some embodiments, after a successful connection, the second device 104C can seamlessly appear along any other image capture source available for selection in the first device 102, such as an image capture device located inside (i.e., built-in) the first device 102 (e.g., image capture device 106), or an image capture device directly connected to the first device 102 (e.g., via a USB port as shown in the example of Scenario 1B).

[0039] An exemplary live video image stream extended in a graphical window containing visual content.

[0040] Referring here to Figure 2, various examples of a live video image stream extended with a graphical window containing visual content are shown according to one or more embodiments. Note that the three scenarios 200A-200C provide examples of views that a receiver of a video conference or other AV content session might see, according to several implementation forms. In other words, the human subject shown in scenarios 2B (200B) and 2C (200C) of Figure 2, also referred to in this example as the “presenter,” is a transmitting party located in a different physical location from the first device 102, and the graphical content 202A-202C reflects the visual content being transmitted by the transmitting party to the receiving party. In some cases, the transmitting party may experience a “preview” window on the display of an electronic device that shows a view similar to that of the receiving party, a view different from that of the receiving party, and / or a preview of the live video image stream being transmitted to the receiving party.

[0041] Starting with Scenario 2A (200A), the so-called “presenter off” mode is demonstrated, where the receiver receives a view of only the graphical window 202A containing visual content (in this case, the capital letter “A” represents the visual content transmitted within the graphical window). In Scenario 2A, the “presenter” (e.g., the transmitting party) is not transmitted to the receiving party. However, in some implementations of Scenario 2A, one or more synchronized audio streams may be transmitted to the receiver along with the graphical window 202A (for example, one audio stream may be obtained from the active microphone on the presenter’s transmitting device, and another audio stream may be associated with the visual content displayed in the graphical window 202A).

[0042] Next, moving to Scenario 2B (200B), a so-called "presenter-small" mode is shown, where the receiving party receives a view 202B of a graphical window containing visual content, as well as a visual representation (204) of the presenter. In some implementations of Scenario 2B, the foreground portion of the image captured by the transmitting device (e.g., including a human subject) may be further cropped based on, for example, the size and location of the detected face of the human subject before being composited, rendered, and sent to the receiving device with the visual content of the graphical window 202B. As shown in Scenario 2B (200B), a relatively small and tight crop around the head of the presenter 204 is created by the transmitting party. In this way, the visual content of the graphical window 202B may remain the focus of the video conferencing session, yet still give the recipient a view and connection to the presenter while they are speaking. In some implementations, an image framing framework implementing one or more predetermined sets of cropping rules may be performed on the transmitting device to maintain the presenter centering, zooming to the appropriate level, distortion correction, etc., so that the presenter 204's head remains well-framed in the composite image transmitted to the receiving side (if desired in a given implementation).

[0043] In some implementations, the sending party may be able to drag or otherwise move a smaller representation of the presenter 204 around the video conferencing session window in real time to rearrange and / or resize the representation relative to the visual content of the graphical window 202B in a visually pleasing manner (see, for example, arrow 206).

[0044] In some cases, a smaller representation of presenter 204 may be rendered at the same depth as the visual content of graphical window 202B, regardless of the depth of presenter 204 in the captured scene. In other cases, a smaller representation of presenter 204 may be rendered at a deeper (or lower) depth than the visual content of graphical window 202B, depending on the preference of a given use case. In yet another implementation of Scenario 2B, the smaller representation 204 of the presenter may be bound to the graphical window 202B shared by the user.

[0045] In yet another implementation of Scenario 2B, one or more synchronized audio streams may be sent to the receiver along with the graphical window 202B (for example, one audio stream may be obtained from the active microphone on the presenter's transmitting device, and another audio stream may be associated with the visual content displayed in the graphical window 202B).

[0046] Finally, moving to Scenario 2C (200C), the so-called “Presenter-Large” mode is shown, in which the receiving party receives a view 202C of a graphical window containing visual content, as well as a visual representation (208) of the presenter. In some implementations of Scenario 2C, the graphical window 202C containing the visual content is assigned a depth value and can be composited in front of (i.e., at a shallower depth), behind (i.e., at a deeper depth), or at the same depth as the foreground portion of the image captured by the transmitting device (e.g., containing a human subject).

[0047] As shown in Scenario 2C (200C), a relatively small view 202C of the graphical window is generated by the sending party's device and positioned behind or "on the shoulders" of the presenter (208). In this way, the presenter (208) can maintain focus on the video conference session while still giving the receiver a view and connection to the visual content of the graphical window.

[0048] In some implementations, the sending party may rearrange and / or resize the representation of the graphical window 202C to the presenter (208) in a visually pleasing manner, i.e., before sending the synthesized and rendered video image stream to the receiving party.

[0049] In yet another implementation of Scenario 2C, one or more synchronized audio streams may be sent to the receiver along with the graphical window 202C (for example, one audio stream may be obtained from the active microphone on the presenter's transmitting device, and another audio stream may be associated with the visual content displayed in the graphical window 202C). It should be understood that the presenter can seamlessly transition between any of Scenarios 2A, 2B, and 2C within the same live video image stream, as desired.

[0050] Examples of visual effects that can be applied to enhance a live video stream

[0051] Referring here to Figure 3A, an exemplary visual effect 300 that may be applied to extend a live video stream according to one or more embodiments is shown. The exemplary visual effect 300 includes placing and compositing one or more graphical elements (306) into the live video image stream. In some embodiments, the graphical elements 306 may include depth-aware graphical representations (e.g., emojis and / or images or a series of images). In other embodiments, the graphical representation may represent the reactions of a human subject in the video image stream (302). In some cases, the reactions of the human subject 302 may be indicated by facial expressions, movements, gestures (e.g., hand gestures 304), spoken words commands, or typed words commands in the video image stream. In some cases, at least one of the depth values ​​or placements of a particular graphical element 306 may be based at least in part on the size or location of the human subject 302.

[0052] Returning to the specific example in Figure 3A, when an exemplary hand gesture 304 is detected by one or more algorithms that process and interpret video images captured in real time by the transmitting device, a series of graphical emoji hearts (3061-3064) are synthesized and rendered into an enhanced version of the original captured video image. In this example, the first heart (3061) is positioned within the image frame, for example, slightly overlapping with and to the left of the detected head of a human subject 302 in the foreground portion of the initially captured video image. Furthermore, the first heart (3061) is assigned a depth value smaller than the determined / estimated depth associated with the human subject in the foreground portion of the scene, so that the first heart (3061) can obscure the view of part of the human subject's head 302. Additional graphical elements may be positioned at the same depth as the head of the human subject 302, for example (3062), or at a deeper depth than the head of the human subject 302, for example (3063 and 3064). Please understand that the example shown in Figure 3A is just one example of a depth-aware visual effect that may be applied to extend a live video stream according to the techniques described herein.

[0053] Furthermore, it should be understood that the arrangement of various graphical elements 306 within the video image may change over time, for example, based on what a particular visual effect is trying to convey. For example, a heart 306 may rotate around the head of a human subject 302 for a fixed amount of time, fade into the background of the scene at a predetermined speed, and eventually disappear, and may change in size / color / transparency over time. However, since graphical elements can perceive depth and recognize the position (and size) of a human subject in the foreground portion of the captured video image, contextually relevant visual effects can be synthesized and rendered into the augmented live video image stream according to the techniques described herein.

[0054] The exemplary hand gesture 304 is just one example of a detected gesture that may trigger the initiation of a visual effect. In other embodiments, gestures that are right-hand specific, left-hand specific, or multi-hand may be detected, and may include specific use of specific fingers or specific sequences of movements. Furthermore, the size and / or placement of a given visual effect may be determined at least in part on the type of visual effect that was initiated. For example, as described below with reference to Figure 3B, some visual effects may occupy (and / or replace) the entire background portion of the captured video image. Other visual effects may be placed based on the current location of the hands, head, body, etc., of a human subject.

[0055] Referring here to Figure 3B, another exemplary visual effect 350 that may be applied to extend a live video stream according to one or more embodiments. As suggested above, in the example of the visual effect 350, an exemplary two-finger gesture 354 made by a human subject 352 (which may also be required to be made within a specific depth range, such as with a particular hand of the user, at a specific distance from the user's head, or within a captured scene) initiates a visual effect that occupies (and replaces) the entire background portion of the captured video image. In some implementations, the visual effect 350 may be played over a predetermined number of captured video image frames or a predetermined amount of time. In some implementations, the visual effect 350 may be based on the successful execution of a segmentation algorithm that identifies the background portion of each captured video image frame in real time. In some implementations, the visual effect 350 may also include introducing "artificial" shadows or other lighting / coloring changes on the face or body of the human subject 352 to provide the receiver with a greater sense of reality that the visual effect is truly "interacting" with the human subject 352, for example.

[0056] Visual effects configured to improve the resolution of visual content transmitted as part of a live video stream.

[0057] Referring here to Figure 4, an exemplary visual effect 400 is shown, in one or more embodiments, configured to improve the resolution of visual content transmitted as part of a live video stream. As shown in Figure 4, an exemplary transmitting electronic device 402 transmits a live video image stream (which may also include one or more streams of synchronized audio data) to exemplary receiving electronic devices 408 and 410.

[0058] The display of the transmitting electronic device 402 may have a first display resolution, for example, 1080p (i.e., 1,080 vertical pixel rows), 4K (i.e., 2,160 vertical pixel rows), 8K (i.e., 4,320 vertical pixel rows), and so on. The display of the transmitting electronic device 402 may also be used to display one or more graphical windows containing visual (or AV) content, such as the exemplary windows 404, 4061, and 4062 shown in Figure 4. In the example shown in Figure 4, the user has chosen to share the content currently displayed in window 404 with the receiving party operating the exemplary receiving electronic devices 408 and 410.

[0059] As illustrated, during a content sharing session, the sending user may display window 404 at a resolution much smaller than full-screen resolution (e.g., within a 300x200 pixel graphical window) for reasons such as the overall screen size limitations on the sending user's device, the simultaneous use of other graphical windows (e.g., 4061 and 4062), or any number of other organizational or aesthetic reasons. In a simple implementation, as shown in the upper half of Figure 4 (labeled "Scenario 4A 'Always HD' Off"), the visual content displayed in the shared window 404 may be backed up by a full-display-size buffer on the 402 device (e.g., a 1080p buffer, 4K buffer, 8K buffer, etc.). However, because the shared window 404 is very small locally on the sending user's device, it would be dramatically upscaled to fill the full-display-size buffer, and the text and / or images may become blurry when displayed at full-screen resolution on the receiving party's device, as shown on the display screen of the exemplary receiving electronic device 408.

[0060] In contrast, in the novel implementation described herein that improves the resolution of visual content transmitted as part of a live video stream, content shared by the transmitting party (e.g., windows, screens, applications, etc.) can be captured in a buffer before being scaled (e.g., downscaled) to any size the transmitting party is currently viewing it on. This buffer information can then be provided as a prerequisite to a capture application (e.g., a video conferencing or AV content sharing application), which can then (if necessary) be scaled down and placed on the display screen of the transmitting party's device at the desired size. In this way, the AV content captured before downscaling, as well as all system fonts, system text, or any other graphical content that is programmatically rendered or independent of resolution, will be sharpest when received by the receiving party, as shown in the lower half of Figure 4 and the exemplary receiving electronic device 410 (labeled “Scenario 4B “Always HD”-ON”). In other words, using these resolution preservation techniques, AV content received by the receiving party as part of an AV sharing session always has an "HD" (i.e., high definition) appearance, that is, independent of the resolution at which the visual content is currently displayed on the transmitting electronic device. In some cases, the resolution of the visual content may be the same on the transmitting and receiving devices; in other cases, the resolution may be higher on the receiving device (with the advantage of no loss of visual clarity, as mentioned above); and in yet other cases, the resolution may actually be lower on the receiving device (if so desired by the receiving party). In any case, the receiving party is no longer constrained or limited by the resolution of the visual content currently displayed on the transmitting party's electronic device.

[0061] In some embodiments, an accelerated experience can be provided to the sending / sharing parties to select content and initiate a content sharing session. In particular, in such embodiments, the visual content / graphical window that the parties wish to share can be conveniently selected via the application programming interface (API) or operating system (OS) level functionality of the sending electronic device. In other words, the screen / content sharing function / menu of individual video conferencing or other AV content sharing applications no longer needs to be used to initiate a content sharing session. Instead, an OS-level function for selecting the content to be shared may be used, which may be incorporated, for example, into any graphical window displayed within the OS or in some other OS-level control panel, as desired by a given implementation. Once the sharing selection is specified and confirmed by the user, the content can begin to be shared with the receiving party in a seamless manner. In some embodiments, additional graphical windows, applications, etc., can be added to the sharing session via the aforementioned OS-level functionality, such as by simultaneously utilizing the content sharing functions of two different video conferencing applications, without having to discard the initial sharing session. In some embodiments, OS-level functionality can also provide the user with the ability to seamlessly "swap" entire graphical windows, applications, displays, etc., that are shared with the recipient during a given content sharing session.

[0062] Virtual lighting effects that can be applied to enhance live video streams

[0063] Referring here to Figure 5, exemplary virtual lighting effects 500 that may be applied to enhance a live video stream in one or more embodiments are shown. Effective and / or interesting lighting of presenters or participants in a video conferencing session can be an important feature to add quality for both the sending and receiving parties and to improve the content sharing experience. However, due to, for example, lack of space, lack of time, lack of equipment, and / or lack of training / knowledge to light themselves (or their environment) in an effective and / or interesting way, it is not always possible for video conferencing participants to adopt advanced studio lighting settings during a content sharing session.

[0064] Therefore, it is desirable to be able to add customizable and intelligent virtual lighting effects to the live video image stream using lightweight and efficient technology in order to enhance the video image stream and provide higher quality images to the receiving party. Ideally, such virtual lighting effects could be applied in a high-performance manner, leveraging existing segmentation / person recognition technologies, notified by state-of-the-art AI / ML-based lighting models, and ultimately synthesized and rendered onto the enhanced video image sent to the receiving party, so that the receiving party does not need any dedicated software, functions, or applications to experience the enhanced live video image stream with the enhanced virtual lighting effects applied.

[0065] According to some embodiments, the virtual lighting effect may be applied only to the head area of ​​an identified human subject (510) in the live video image stream and / or to a specific background surface (e.g., a wall 506). This intentional limitation on the area of ​​the captured scene to which the virtual lighting effect is applied can help to make the application of the virtual lighting effect more efficient and / or to avoid applying the virtual lighting effect to areas of the captured scene where confidence or knowledge of the depth and / or geometric structure of surfaces and objects in the scene (e.g., background, walls, flat surfaces, dimly lit areas of the scene, inanimate objects, etc.) is lower. Of course, if there is sufficient time, surface shape confidence, and / or processing / thermal resources available to the transmitting electronic device, the virtual lighting effect described herein can also be applied to the entire captured video image frame.

[0066] According to some embodiments, the virtual lighting effect may utilize a high-quality real-time map of surface normals (for example, provided by one or more AI / ML-based or other image processing algorithms). Using such a surface normal map, it is possible to calculate and determine the relationship between the surface of the object at that pixel location and the virtual light source that the user is attempting to enhance the captured video image, for each relevant pixel in the video image (for example, any pixel related to the head of a human subject detected in the image). This allows the virtual lighting effect to dynamically track the user's movement and / or changes in the scene over time, and therefore appear more physically accurate than, for example, a static luminance filter being applied to all pixels in a particular part of the captured video image.

[0067] Returning to Figure 5, an exemplary picker user interface (UI) tool 502 is shown, which can provide the user with an array (504) of customizable colors, intensities, angles, positions (e.g., specifying the x, y, and / or z positions of the virtual light sources in the scene), and the number of virtual light sources, which the user may wish to use, in order to enhance the lighting of the captured video image. For example, in Figure 5, the user has chosen to apply a gradient lighting effect that appears to be coming from the left side of the image frame. Thus, only the left side 508 of the face of the human subject 510 and the left wall 506 of the image frame are visually affected by the selected virtual lighting effect. As can be understood, the color, intensity, and / or angle of the virtual lighting effect may be changed or panned, and the resulting effect is applied in real time to the enhanced live video image stream and then transmitted to the receiving party. Also, depending on how much customizability or freedom it is desirable to give the user regarding the selection or number of virtual lighting effects to be applied, any suitable picker UI tool may be used, and it should be understood that the picker UI tool 502 is merely one possible example.

[0068] Exemplary techniques for performing improved audiovisual (AV) synchronization in extended live video streams.

[0069] Figure 6A shows an exemplary technique 600 for performing improved audiovisual AV synchronization in an extended live video stream, according to one or more embodiments. In a typical video conferencing session, the presenter may be streaming their microphone (602), camera 604, graphical window content (606), and / or graphical window audio stream (608) from a local client device (616), as shown in Figure 6A. Thus, there are at least four AV content streams that may need to be time-synchronized before being received by a video conferencing participant (e.g., remote client device 618) (i.e., the T of microphone 602).MIC 601, Camera 604 T CAM 603, T of graphical window 606 WIN 609, and the T of the graphical window audio stream 608 AUD 611). The manner of synchronization is important to prevent “lip-sync” issues between audio and video content for each pair of streams (i.e., AV content streams 601 / 603 associated with the presenter and AV content streams 609 / 611 associated with the presenter’s shared graphical content).

[0070] In some implementations, AV content streams 601 / 603 (related to the presenter) may first be synchronized to a common timeline referred to herein as T1(607) before being combined with other AV content streams (605), and AV content streams 609 / 611 (related to the presenter's shared graphical content) may also first be synchronized to a common timeline referred to herein as T2(615) before being combined with other AV content streams (613).

[0071] Timelines T1 and T2 may differ from each other for various reasons. For example, the user may be using externally connected camera (604) and / or microphone (602) sources, which may have a significant delay such that T2 > T1. This timing difference may not be significant in most use cases, as long as the receiving party can synchronize camera 604 with microphone 602 at time T1 (617) and graphical window content 606 with graphical window audio stream 608 at time T2 before sending them to the remote client device 618.

[0072] However, synchronization problems can arise in extended live video image streams where multiple sources of AV content can be combined together into the same extended output video image. In particular, if the visual effects include streaming a video image that includes presenter composition alongside their screen-sharing content (for example, as described above with reference to scenario 2C200C in Figure 2), the presenter may simultaneously stream three AV content streams: microphone 602, camera / graphical window composition stream 612 as shown in Figure 6A, and graphical window audio stream 608.

[0073] The camera / graphical window composite stream 612 and the graphical window audio stream 608 must both be synchronized to the microphone 602 to prevent undesirable lip-sync issues. In such a case, T1 must drive the synchronization process to avoid lip-sync issues, so the receiving party cannot use capture time T2(615) to synchronize the graphical window content 606 and the graphical window audio stream 608 with the microphone 602. Similarly, simply adding T2 to the composite stream 612 would consume additional network bandwidth, which is undesirable. Furthermore, the payload of the Real-Time Transport Protocol (RTP) standard does not allocate sufficient space to transmit the streams at both T1 and T2.

[0074] Accordingly, a novel technique for ensuring that the graphical window content 606 in the composite stream 612 is precisely synchronized with the graphical window audio stream 608 of the receiving party is described herein. In particular, the graphical window content 606 may be captured by a screen capture application (610) on a local client device 616, i.e., still using the value of T2 (626), before being composited into the composite stream 612, which will be configured to use T1 (630).

[0075] According to some embodiments, a moving average (614) can be calculated for the time difference between T2 and T1, while the original value of T1 (628) is still passed to the composite stream 612 of camera 604. The time of the original window audio stream 608 (i.e., the time at T2) may then be replaced with the value of T1' (632), where T1' may be calculated as T2 minus the moving average difference calculated in 614, as described above.

[0076] Finally, the receiving party (e.g., remote client device 618) can synchronize the window audio stream 608 with the microphone stream using the value of T1' (indicated as 642 in remote client device 618) (indicated by the dashed line 636 connecting stream 638 and stream 642 in remote client device 618). Similarly, the receiving party can synchronize the composite stream 612 with the microphone stream using the value of T1 (indicated as 640 in remote client device 618) (indicated by the dashed line 634 connecting stream 638 and stream 640 in remote client device 618).

[0077] Figure 6B shows another exemplary technique 650 for performing improved AV synchronization in an extended live video stream, according to one or more embodiments. (Note: Element numbers common between Figure 6A and Figure 6B refer to the same or similar system components and are therefore not described in detail again with reference to Figure 6B.)

[0078] In other types of extended live video image streams, the visual effects may include a streaming of video images alongside (or on top of) the selected screen-sharing AV content, which includes a composite screen capture of a movable, smaller / cropped representation of the presenter (e.g., only the presenter's head) captured by camera 604 (as described above with reference to scenario 2B200B in Figure 2, for example). Here again, the presenter in such a scenario may stream at least three AV content streams simultaneously, namely microphone 602, a screen capture stream 610 (i.e., reflecting a smaller / cropped output from camera 604, which is composited with the graphical window of the selected screen-sharing AV content), and a graphical window audio stream 608, as shown in Figure 6B.

[0079] According to some embodiments, to achieve a more computationally efficient design, a smaller / cropped representation of the presenter may first be combined with a graphical window of selected screen-sharing AV content and then displayed on the presenter's screen on the local client device 616. Next, the entire screen (or a portion of the screen) can be captured by a screen capture framework (654) and output as a screen capture stream 610 containing the smaller / cropped representation of the presenter along with the selected screen-sharing AV content. Similar to the example shown in Figure 6A, in this configuration 650 shown in Figure 6B, at least three ATV content streams are sent to the remote client device 618.

[0080] In this configuration 650 shown in Figure 6B, a new technique is introduced to prevent the loss of T1 information necessary for the receiving party to synchronize the camera 604 with the microphone 602. First, the value of T1 (660) may be attached to the composite camera frame sent to the display rendering server of the local client device 616, and the local client device 616 may then display the composite camera frame on the presenter's screen (for example, in the screen capture preview block 652).

[0081] Next, the rendering server of the screen capture framework 654 may detect the value of T1_attach received from the screen capture preview block 652 (662), and may attach this information to the screen capture frame captured by the screen capture framework 654 at time T2 (615) (i.e., the graphical content that the presenter wishes to share). The screen capture framework 654 then replaces the screen frame time T2 (615) with the T1_attach rendering server (662) time and calculates the moving average difference between T2 and T1_attach at 656, but the screen capture stream 610 may still be sent to the remote client device 618 using the rendering server (658). Furthermore, the time T2 (615) of the graphical window audio stream 608 may be replaced with the value of T1' (666), which may be calculated as T2 minus the moving average difference calculated at 656.

[0082] Finally, the receiving party (e.g., remote client device 618) can synchronize the window audio stream 608 with the microphone stream using the value of T1' (indicated as 668 in remote client device 618) (indicated by the dashed line 670 connecting stream 638 and stream 668 in remote client device 618). Similarly, the receiving party can synchronize the screen capture stream 610 with the microphone stream using the value of T1_attach (indicated as 664 in remote client device 618) (indicated by the dashed line 672 connecting stream 638 and stream 664 in remote client device 618).

[0083] As can be understood here, the technique shown in Figure 6B thus provides the receiving party with the information necessary to synchronize the microphone stream, screen capture stream, and window audio stream of the transmitting device.

[0084] An exemplary method for augmenting a live video image stream using depth-aware visual effects.

[0085] Figure 7 is a flowchart of Method 700 for extending a live video image stream using depth-aware visual effects, according to various embodiments. First, in step 702, Method 700 may acquire a video image stream containing multiple images of a scene captured by a first image capture device in a first electronic device. Next, Method 700 may start a for loop 704 through which steps 706-716 can be iteratively performed for each of one or more images in the video image stream. Steps 706-716 are described herein in the context of being performed for a “first image” of the video image stream, but it should be understood that one or more of steps 706-716 can be similarly performed for any number of images in the video image stream to which depth-aware visual effects are desired.

[0086] Referring next to step 706, with respect to at least a first image of the video image stream, method 700 can perform a segmentation operation on the first image to identify at least a foreground and background portion of the first image. As described above, in some embodiments, the segmentation operation may be enabled by one or more artificial intelligence (AI) or machine learning (ML) based algorithms, for example, an AI / ML based algorithm trained to identify and segment out foreground human subjects (and / or common handheld objects or other accessories that may be held by or otherwise connected to such foreground human subjects) in the captured image. Preferably, such a segmentation operation is high-performance (e.g., in terms of both processing and thermal efficiency) and can deliver an accurate segmentation mask for the captured image in real time, i.e., as the captured video image is being streamed from the image capture device.

[0087] Next, in step 708, method 700 may assign a first depth value in the scene to the foreground portion of the first image. As described above, in some embodiments, the first depth value assigned to the foreground portion of the first image may be the actual estimated depth of a human subject (or other object) that constitutes the majority of the determined foreground portion of the first image. For example, the first depth value may be estimated via stereo imaging, disparity calculation, time-of-flight (ToF) camera, phase detection pixels, AI / ML-based monocular depth estimation framework, or other depth estimation modalities that may be preferred or available in a given implementation. In other embodiments, the first depth value assigned to the foreground portion of the first image may be a depth different from the estimated actual depth of a human subject in the foreground portion of the first image. For example, in the extended output image, the foreground portion of the captured first image may be modified to appear to have a closer (or further) depth in the first image, or the foreground portion may be cropped and composited with the same apparent depth (or a different depth) as other visual content that may be composited with the foreground portion before being transmitted to a second electronic device, for example.

[0088] Next, in step 710, method 700 may optionally assign a second depth value in the scene to the background portion of the first image. As described above, the background portion of the first image may be estimated using segmentation behavior (for example, in some embodiments, any portion of the first image not designated as a foreground portion may be designated as a background portion). In some embodiments, the actual estimated depth of the background portion of the first image may be determined via any preferred depth estimation modality. In other embodiments, any other value may simply be assigned to function as the second depth value of the background portion of the first image. In yet another embodiment, no value may be used (or a default value may be used) to function as the second depth value of the background portion of the first image. As described above, some visual effects may require scene background depth, and as a result, depth-aware visual effects may be rendered with the correct apparent depth in the scene relative to the foreground portion of the scene and / or have appropriate / semantically correct interaction with the assigned scene background. In some implementations, the estimated second depth value of the scene background can also assist the segmentation operation by allowing it to tighten (or loosen) its constraints when identifying which parts of the captured image are likely to be part of the foreground.

[0089] Next, in step 712, method 700 can extend the first image with at least a first visual effect, the first visual effect being assigned a third depth value in the scene. Many different types of depth-aware visual effects are possible to extend a live video image stream, as described above with reference to Figures 2, 3, and 5, and those described herein are only a few examples. As described above, by assigning each visual effect its own depth value in the scene, more realistic composite rendering can be performed, including, for example, interaction and / or context recognition between the visual effect and an object detected in the foreground portion of the captured image (e.g., a human subject).

[0090] Next, in step 714, method 700 may combine at least (a) the foreground portion of the first image at a first depth value and (b) the rendering of the first visual effect at a third depth value into the first extended output image. In some embodiments, all or part of the background portion of the first image may also be included in the first extended output image, for example, at a second depth value, if 1 is assigned in step 710. For example, by combining at least two layers of the foreground portion and the visual effect into an extended output image before being transmitted to a second electronic device (the extended output image may be combined with other such extended output images over time into an extended video image stream), such a second electronic device does not require any specialized software or recognition of special effects protocols or formats to correctly render the visual effect on the second electronic device. Instead, the visual effect is already "burned into" the video, which can be encoded and / or transmitted according to any standardized or desired video encoding format.

[0091] Finally, in step 716, method 700 may transmit the first extended output image to a second electronic device. As described above, this transmission may be performed as part of a video image stream being transmitted to the second electronic device according to any standardized or desired video transmission protocol, for example, as part of a video conferencing application, a screen sharing application, etc.

[0092] To make it clear, for example, with reference to Figure 7, the various methods described herein may be performed by an electronic device, for example, by being initiated by an application (or “App”) running on the device and / or the device’s native operating system (OS). For example, an App running on a device may initiate or perform all or at least some of the steps in the Method, while calling the device’s OS to perform other steps in the Method. Similarly, the device’s OS may receive API calls from an App or elsewhere, process / execute those calls, and allow the device to perform the Method.

[0093] Exemplary electronic computing devices

[0094] Referring here to Figure 8, a simplified functional block diagram of an exemplary programmable electronic computing device 800 according to one embodiment is shown. The electronic device 800 may be, for example, a mobile phone, a personal media device, a portable camera, or a tablet, notebook, or desktop computer system. As shown, the electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity sensor / ambient light sensor, accelerometer, inertial measurement unit, and / or gyroscope), a microphone 830, an audio codec 835, a speaker 840, a communication circuit 845, an image capture device 850, a video codec 855, a memory 860, a storage device 865, and a communication bus 870, which may include, for example, multiple camera units / optical image sensors having different features or capabilities (e.g., still image stabilization (SIS), HDR, OIS system, optical zoom, digital zoom, etc.).

[0095] The processor 805 can execute instructions necessary to perform or control the operation of numerous functions performed by the electronic device 800 (e.g., the generation, processing, and / or streaming of image and video data according to various embodiments described herein). The processor 805 can, for example, drive the display 810 and receive user input from the user interface 815. The user interface 815 can take various forms such as buttons, keypads, dials, click wheels, keyboards, display screens, and / or touch screens. The user interface 815 can, for example, be a conduit through which a user can view a captured video stream and / or indicate a specific image frame that the user wants to capture (e.g., by clicking a physical or virtual button at the moment a desired image frame appears on the device's display screen). In one embodiment, the display 810 can display a video stream when the video stream is being captured while the processor 805 and / or graphics hardware 820 and / or image capture circuitry are simultaneously generating and storing the video stream in memory 860 and / or storage device 865. The processor 805 may be a system-on-a-chip (SOC), such as those found in mobile devices, and may include one or more dedicated graphics processing units (GPUs). The processor 805 may be based on a reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architecture, or any other suitable architecture, and may include one or more processing cores. The graphics hardware 820 may be dedicated computing hardware for processing graphics and / or for assisting the processor 805 in performing computational tasks.In one embodiment, the graphics hardware 820 may include one or more programmable graphics processing units (GPUs) and / or one or more specialized SOCs, such as Apple's Neural Engine processing core, which are specially designed to perform neural network and machine learning computations (convolutions) in a more energy-efficient manner than either the main device's central processing unit (CPU) or a typical GPU.

[0096] The image capture device 850 may comprise one or more camera units configured to capture images, for example, images that can be processed to produce cropped, enlarged, and / or distortion-corrected versions of the captured images as described herein. The image capture device 850 may comprise two (or more) lens assemblies 880A and 880B, each having a distinct focal length. For example, lens assembly 880A may have a shorter focal length than lens assembly 880B. Each lens assembly may have a distinct associated sensor element, for example, sensor element 890A / 890B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture device 850 can capture still images and / or video images. The output from the image capture device 850 can be processed, at least in part, by the video codec 855 and / or the processor 805 and / or the graphics hardware 820 and / or a dedicated image processing unit or image signal processor incorporated within the image capture device 850. The captured images can then be stored in the memory 860 and / or the storage device 865.

[0097] Memory 860 may include one or more different types of media used by the processor 805, graphics hardware 820, and image capture device 850 to perform the functions of the device. For example, memory 860 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and other appropriate data. Storage device 865 may include one or more non-temporary storage media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). Memory 860 and storage device 865 may be organized into one or more modules and used to hold computer program instructions or code written in any desired computer programming language. For example, when executed by processor 805, such computer program code can perform one or more of the methods or processes described herein. Power supply 875 may include a rechargeable battery (e.g., a lithium-ion battery) or other electrical connection to a power supply, such as a mains power supply, used to manage and / or provide power to the electronic components and associated circuits of electronic device 800.

[0098] It should be understood that the above description is illustrative and not limiting. For example, the embodiments described above can be used in combination with each other. A number of other embodiments will become apparent to those skilled in the art when considering the above description. Therefore, the scope of the present invention should be determined by referring to the appended claims and the entire scope of equivalents given to such claims.

Claims

1. It is a method, In a first electronic device, a video image stream comprising multiple images of a scene captured by a first image capture device is acquired, For at least the first image of the aforementioned video image stream, Performing a segmentation operation on the first image to identify at least the foreground and background portions of the first image, Assigning a first depth value within the scene to the foreground portion of the first image, Assigning a second depth value within the scene to the background portion of the first image, At least a first visual effect, the first visual effect being assigned a third depth value in the scene, extends the first image with at least the first visual effect, At a minimum, (a) the foreground portion of the first image at the first depth value, and (b) the rendering of the first visual effect at the third depth value are combined into the first extended output image. A method comprising transmitting the first extended output image to a second electronic device.

2. The method according to claim 1, wherein the first image of the video image stream is visually enhanced by at least one of the following before being acquired by the first electronic device: trimming according to one or more predetermined framing rules, applying distortion correction, or applying tone mapping.

3. The method according to claim 1, wherein the first image capture device is connected to the first electronic device by one of the following means: via a wired connection, a wireless connection, or by being incorporated into the first electronic device.

4. The method according to claim 1, wherein the first extended output image is transmitted to the second electronic device as part of a video conferencing application.

5. The method according to claim 1, wherein the foreground portion of the first image includes at least one human subject.

6. The method according to claim 1, wherein the first depth value is assigned based on the estimated depth of the foreground portion of the first image within the scene.

7. The third depth value is, (1) Smaller than the first depth value, (2) A value greater than the first depth value, (3) Between the first depth value and the second depth value, (4) The method according to claim 1, wherein at least one of the following is equal to the first depth value.

8. The first visual effect described above is (1) Virtual lighting effect, (2) A graphical window containing visual content, or The method according to claim 1, further comprising at least one of (3) a graphical representation of the reaction of a human subject captured in a scene.

9. The method according to claim 1, wherein the first visual effect includes a graphical window containing visual content, and the third depth value is greater than the first depth value.

10. The method according to claim 5, wherein the first visual effect includes a graphical window containing visual content, the foreground portion being cropped before rendering based on the size and location of the detected face of the human subject.

11. The method according to claim 5, wherein the first visual effect includes a graphical representation of the human subject's reaction, and at least one of (1) the third depth value, or (2) the arrangement of the graphical representation of the human subject's reaction, is at least partially based on the size or location of the human subject.

12. The method according to claim 5, wherein the first visual effect includes a graphical representation of the human subject's reaction, and the graphical representation includes emojis or images.

13. The method according to claim 1, wherein the first visual effect includes a graphical window containing visual content, the visual content of the graphical window is selected via an application programming interface (API) or operating system (OS) level function of the first electronic device.

14. The method according to claim 4, wherein the first visual effect includes a graphical window containing visual content, the visual content being rendered, at least partially, to a first extended output image at a second resolution independent of the first resolution at which the visual content is displayed on the first electronic device.

15. The method according to claim 5, wherein the first visual effect includes a virtual illumination effect, the virtual illumination effect includes at least one of (1) a virtual light color, (2) a virtual light intensity, or (3) a virtual light arrangement, and extending the first image with the virtual illumination effect further includes estimating a set of surface normals for the human subject in the first image.

16. At least a second visual effect, the second visual effect further includes extending the first image with at least the second visual effect to which a fourth depth value in the scene is assigned, The aforementioned synthesis further includes (c) compositing the rendering of the second visual effect at the fourth depth value into the first extended output image, The third depth value and the fourth depth value are different, The method according to claim 1.

17. The method according to claim 1, wherein the first visual effect includes a graphical window containing first audiovisual (AV) content, the video image stream includes second AV content, and the timing of the audio components of the first AV content is adjusted at least in part based on the timing of the audio components of the second AV content before being transmitted to the second electronic device.

18. The method according to claim 17, wherein the timing of the audio component of the first AV content is further adjusted at least in part on a moving average difference between (1) the timing of the audio component of the first AV content and (2) the timing of the audio component of the second AV content.

19. It is an electronic device, Memory and The system comprises one or more processors operably coupled to the memory, wherein the one or more processors are connected to the one or more processors, An electronic device configured to execute an instruction causing the method according to any one of claims 1 to 18.

20. A non-temporary computer-readable medium (CRM) containing computer-readable instructions, wherein the computer-readable instructions are processed by one or more processors. A non-temporary computer-readable medium (CRM) on which it is possible to carry out the method according to any one of claims 1 to 18.

Images