System and method for multi-user video communication with partition detection and adjustable fiximity

Abstract

In one embodiment, a method includes maintaining video communication between client devices, each client device associated with a respective video stream associated with a respective matching scale based on a height of a frame of the respective video stream and a depth measurement of an object within the frame; a respective scaling factor and a respective clip are determined for each video stream, where the respective scaling factor is calculated based on the matching scale associated with the respective video stream and a target matching scale determined from matching scales associated with all video streams, and where the respective scaling factor is calculated based on the matching scale associated with the respective video stream and the target matching scale is calculated based on the matching scales associated with all video streams. The respective cropping is determined based on consistency across positions of the objects within the frames of all video streams; a merged video stream is generated from each video stream based on the respective scaling factor and the respective cropping, and an instruction to present the merged video stream is sent to one or more of the client devices.

Description

Technical Field

[0001] The present disclosure relates generally to database and file management within a network environment, and particularly to video communications. Background Art

[0002] Standard video call systems are common. They have fixed start and end times, and during this time, they transmit video and audio between participants (which can be enabled or disabled). They are often used between friends, family members, business calls (one-to-one), business meetings (groups), and performances (one-to-many). There are many different platforms with different features customized for use cases, such as replacing a person's video with an avatar on a friend-centric platform, applying noise suppression in a game-centric platform, or applying a virtual background to enhance privacy in a business-centric platform. Some new always-on video communication systems are emerging, which aim to avoid the standard video call start and end structure. These systems are mainly aimed at collaborative workplaces with the goal of reducing barriers between colleagues communicating. Summary of the invention

[0003] Technical Solution

[0004] One embodiment provides a method for maintaining video communication between two or more client devices by an electronic device, wherein each client device is associated with a corresponding video stream in the video communication, and wherein each video stream is associated with a corresponding matching scale based on a height of a frame of each video stream and a depth measurement of an object within the frame. One embodiment provides a method for determining a corresponding scaling factor and a corresponding cropping by an electronic device for each of the video streams, wherein the corresponding scaling factor is calculated based on a matching scale associated with the corresponding video stream and a target matching scale, the target matching scale being determined from matching scales associated with video streams associated with the two or more client devices, and wherein the corresponding cropping is determined based on consistency of positions of objects within frames across the video streams associated with the two or more client devices. One embodiment provides a method for generating a merged video stream from each of the video streams for video communication by an electronic device based on the corresponding scaling factor and the corresponding cropping of each video stream. One embodiment provides a method for sending an instruction to one or more of the client devices for presenting the merged video stream by an electronic device.

[0005] One embodiment provides one or more displays; one or more non-transitory computer-readable storage media, including instructions; and one or more processors, coupled to the storage media, the one or more processors configured to execute the instructions to: maintain video communications between two or more client devices, wherein each client device is associated with a corresponding video stream in the video communications, and wherein each video stream is associated with a corresponding matching scale based on a height of a frame of each video stream and a depth measurement of an object within the frame. The one or more processors are also configured to execute the instructions to determine a corresponding scaling factor and a corresponding cropping for each of the video streams, wherein the corresponding scaling factor is calculated based on the matching scale associated with the corresponding video stream and a target matching scale, the target matching scale being determined from the matching scale associated with the video streams associated with the two or more client devices, and wherein the corresponding cropping is determined based on consistency of positions of objects within the frames across the video streams associated with the two or more client devices. The one or more processors are further configured to execute the instructions to generate a merged video stream from each of the video streams for the video communication based on the respective scaling factors and the respective crops of each video stream. The one or more processors are further configured to execute the instructions to send instructions for presenting the merged video stream to one or more of the client devices.

[0006] One embodiment provides a computer-readable non-transitory storage medium including instructions executable by a processor to: maintain video communications between two or more client devices, wherein each client device is associated with a respective video stream in the video communications, and wherein each video stream is associated with a respective matching scale based on a height of a frame of each video stream and a depth measurement of an object within the frame. The instructions, when executed by the at least one processor, further determine a respective scaling factor and a respective cropping for each of the video streams, wherein the respective scaling factor is calculated based on the matching scale associated with the respective video stream and a target matching scale determined from the matching scale associated with the video streams associated with the two or more client devices, and wherein the respective cropping is determined based on consistency of positions of objects within frames across the video streams associated with the two or more client devices. When executed by the at least one processor, the instructions further generate a merged video stream from each of the video streams for the video communications based on the respective scaling factor and the respective cropping of each video stream. The instructions, when executed by the at least one processor, also send instructions to one or more of the client devices for presenting the merged video stream.

[0007] These and other aspects and advantages of one or more embodiments will become apparent from the following detailed description which, when taken in conjunction with the accompanying drawings, illustrates, by way of example, the principles of one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example electronic device is shown.

[0009] Figure 2 An example comparison between high-fidelity video and low-fidelity video is shown.

[0010] Figure 3 An example flow chart for determining a level of engagement at a particular device is shown.

[0011] FIG. 4A to FIG. 4B An example viewing angle adjustment is shown.

[0012] Figure 5 An example process for calculating a scaling factor for each displayed video stream is shown.

[0013] Figure 6 Example assumptions for the length of visible content are shown.

[0014] Fig. 7A Example measurements of vertical distance at an object are shown.

[0015] Figure 7B An example visualization of vertical distance at an object is shown.

[0016] Figure 8 An example zoom of the image is shown.

[0017] Fig. 9 An example crop of the image is shown.

[0018] Fig.10 An example block diagram for adjusting color and lighting is shown.

[0019] Fig.11 An example frame with color and lighting adjustments is shown.

[0020] Fig.12 An example block diagram for implementing image unification is shown.

[0021] Fig.13 An example blend of boundaries is shown.

[0022] FIG. 14A to FIG. 14C An example scenario is shown where the stream size may be adjusted for a better group call experience.

[0023] FIG. 15A to FIG. 15C An example interactive video overlay control is shown.

[0024] Fig.16 An example application of a video communication system is shown.

[0025] Fig.17 An example process of upcycling is shown.

[0026] Fig.18 A flow chart of a method for improving presentation of multiple video streams in video communication according to an embodiment of the present disclosure is shown.

[0027] Fig.19 An example computer system is shown. DETAILED DESCRIPTION

[0028] Mobile Client System Overview

[0029] Figure 1 An example electronic device 100 is shown. According to an embodiment of the present disclosure, the electronic device 100 may include, for example, any of a variety of personal electronic devices 102, such as a mobile phone electronic device, a tablet computer electronic device, a laptop computer electronic device, etc. According to an embodiment of the present disclosure, as shown by Figure 1As further depicted, the personal electronic device 102 may include, among other things, one or more processors 104, memory 106, sensors 108, cameras 110, displays 112, input structures 114, network interfaces 116, power supplies 118, and input / output (I / O) interfaces 120. It should be noted that Figure 1 This is merely one example of a particular implementation and is intended to illustrate the types of components that may be included as part of the electronic device 100 .

[0030] According to an embodiment of the present disclosure, one or more processors 104 may be operably coupled to a memory 106 to execute various algorithms, processes, or functions. Such programs or instructions executed by the processor 104 may be stored in any suitable article of manufacture, which includes one or more tangible computer-readable media that at least collectively store instructions or routines, such as the memory 106. The memory 106 may include any suitable article of manufacture for storing data and executable instructions, such as a random access memory (RAM), a read-only memory (ROM), a rewritable flash memory, a hard disk drive, etc. In addition, a program (e.g., an operating system) encoded on such a computer program product may also include instructions that can be executed by one or more processors 104 to enable the electronic device 100 to provide various functions.

[0031] According to an embodiment of the present disclosure, the sensor 108 may include, for example, one or more cameras (e.g., depth cameras), touch sensors, microphones, motion detection sensors, heat detection sensors, light detection sensors, time of flight (ToF) sensors, ultrasonic sensors, infrared sensors, or other similar sensors that can be used to detect various user inputs (e.g., user voice input, user gesture input, user touch input, user instrument input, user motion input, etc.). The camera 110 may include any number of cameras (e.g., wide-angle cameras, narrow cameras, telephoto cameras, ultra-wide-angle cameras, depth cameras, etc.) that can be used to capture various 2D and 3D images. The display 112 may include any display architecture (e.g., AMLCD, AMOLED, micro LED, etc.), which may provide further means by which a user can interact and interact with the electronic device 100. According to an embodiment of the present disclosure, such as Figure 1As further shown, one or more of the cameras 110 may be disposed behind, below, or beside the display 112 (e.g., one or more of the cameras 110 may be partially or completely hidden by the display 112), and thus the display 112 may include transparent pixel regions and / or semi-transparent pixel regions through which one or more hidden cameras 110 may detect light and, by extension, capture images. It should be understood that one or more of the cameras 110 may be disposed anywhere behind or below the display 110, such as at a central region behind the display 110, at an upper region behind the display 110, or at a lower region behind the display 110.

[0032] According to an embodiment of the present disclosure, the input structure 114 may include any physical structure for controlling one or more global functions of the electronic device 100 (e.g., pressing a button to "power on" or "power off" the electronic device 100). The network interface 116 may include, for example, any number of network interfaces suitable for allowing the electronic device 100 to access and receive data through one or more cloud-based networks (e.g., cloud-based services that can serve hundreds or thousands of electronic devices 100 and their corresponding associated users) and / or distributed networks. The power supply 118 may include any suitable power source, such as a rechargeable lithium polymer (Li-poly) battery and / or an alternating current (AC) power converter that can be used to power and / or charge the electronic device 100 for operation. Similarly, an I / O interface 120 may be provided to allow the electronic device 100 to be connected to various other electronic or computing devices (such as, one or more auxiliary electronic devices).

[0033] Multi-user video communication with participation detection and adjustable fidelity

[0034] Video calls have been increasingly used. They often have the overhead of previous scheduling via text-based chat services, which may make them prefer longer, more formal, less frequent conversations that expect full participation from start to finish. Therefore, the video call format may be more suitable for some use cases than other use cases. Use cases where existing video call formats are quite flawed can enable natural interactions that occur between people sharing a common physical space, such as interactions between family members sharing a residence. Interactions between close connections in a shared space may mainly include short conversations, accidental encounters, and shared presence. Existing video call systems may not be optimized for short conversations, may not enable accidental encounters, and may not convey a sense of shared presence. One might suggest that a solution to recreate the experience of sharing a space with someone could be to establish a permanently connected video call. However, this approach may introduce many problems. First, unwanted noises such as lawn mowers, kitchen mixers, etc. may be transmitted. Second, conversations occurring in different rooms may be transmitted unintentionally. Third, users may be concerned about oversharing and lack of privacy. Fourth, cameras and microphones may need to be adjusted according to the area of ​​interest. Finally, too much data may be transmitted over the user's Internet connection. To address the above-mentioned issues regarding permanent video calls, the video communication system disclosed herein can always maintain connections between spaces, encourage interaction and convey a sense of presence between users without the disadvantages that would otherwise be associated with permanently connected video calls.

[0035] According to an embodiment of the present disclosure, a video communication system may adjust the fidelity or blurriness of a video communication in order to adjust the connectivity of a user. The video communication system may also determine the user's participation in the video communication by analyzing the video and audio. For example, an enterprise-centric communication platform may implement a video communication system that may detect when someone leaves their seat and, in response, mute their microphone and blur their video. The video communication system may further improve the presentation of multiple video streams by filtering, transforming, and positioning the multiple video streams to appear as a unified scene and automatically adjusting the relative video stream sizes based on the content.

[0036] According to an embodiment of the present disclosure, the video communication system may include one or more cameras, one or more microphones, and one or more sensors to achieve the desired results. As an example and not by way of limitation, the sensor may include a three-dimensional (3D) or depth sensor (e.g., a stereo camera, a lidar sensor, an ultrasonic depth sensor, etc.). The user may issue commands to the video communication system via one or more of a remote control, an application, a voice command, a gesture control, an overlay control, or a home automation control system.

[0037] According to an embodiment of the present disclosure, an electronic device may maintain video communication between two or more client devices. Each client device may be associated with a corresponding video stream in the video communication. Each video stream may be associated with a corresponding matching scale based on the height of the frame of the video stream and the depth measurement of the object within the frame.

[0038] According to an embodiment of the present disclosure, the electronic device may determine a corresponding scaling factor and a corresponding cropping for each video stream. The corresponding scaling factor may be calculated based on a matching scale associated with the corresponding video stream and a target matching scale determined from matching scales associated with the video streams associated with the two or more client devices. The corresponding cropping may be determined based on consistency of positions of objects within frames across the video streams associated with the two or more client devices.

[0039] According to an embodiment of the present disclosure, the electronic device can generate a merged video stream from each video stream in the video communication based on the corresponding scaling factor and the corresponding cropping of each video stream. The electronic device can also send an instruction for presenting the merged video stream to one or more of the client devices.

[0040] There are certain technical challenges for improving the presentation of multiple video streams in video communications. One technical challenge may include effectively detecting the participation level of participants. A solution presented by the embodiments disclosed herein to solve this challenge may be to detect the participation level from inputs including presence and position, body and head direction, gestures, voice, direction of voice arrival, etc., because these inputs provide comprehensive information that can be used to classify each video stream as the correct participation level. Another technical challenge may include accurately calculating the scaling factor for each video stream. A solution presented by the embodiments disclosed herein to solve this challenge may be to calculate the scaling factor based on the vertical distance at the object (VDAS), which indicates the total height of the video frame in meters at the depth of the object (person) within the video frame, because the resulting scaling factor enables the video communication system to scale each image so that the true distance of each pixel (at the depth of the object) can be the same value for all video streams, thereby giving the impression that the object is at the same distance from the camera in a unified scene.

[0041] Certain embodiments disclosed herein may provide one or more technical advantages. The technical advantages of an embodiment may include avoiding oversharing and privacy protection, because the video communication system may adaptively adjust the fidelity or blurriness of the video stream in the video communication based on the connectivity of the corresponding participants. Another technical advantage of an embodiment may include presenting the video streams as a unified scene after the video communication system scales, filters, transforms and positions multiple video streams in the video communication. Another technical advantage of an embodiment may include automatically adjusting the relative video stream size according to the content, because the video communication system may allocate appropriate space for each video stream based on participant count, participation and activity level. Certain embodiments disclosed herein may not provide the above-mentioned technical advantages, provide some or all of the above-mentioned technical advantages. In view of the drawings, description and claims of the present disclosure, one or more other technical advantages may be obvious to those skilled in the art.

[0042] According to an embodiment of the present disclosure, the video communication system may determine the corresponding fidelity for each video stream in the video stream based on the date and time associated with the corresponding video stream, the calendar of the participant associated with the corresponding video stream, the command from the participant associated with the corresponding video stream, or one or more of the degree of participation of the participant associated with the video stream. Therefore, generating a merged video stream can also be based on the corresponding fidelity associated with each video stream in the video stream. The video communication system can transmit and present video and audio with varying fidelity to adjust how well the user connects. The fidelity can be selected by the user, or can be automatically determined using a series of video and audio analysis techniques. Compared with existing video calls, high-fidelity communication can include clear video and clear audio. Lower-fidelity audio can be blurred to different degrees (e.g., muted), visually represented, or completely removed. Lower-fidelity video can be blurred to different degrees (e.g., blurred) or completely replaced by visualization. When the user is fully involved and actively communicates via the system, high-fidelity communication may be most appropriate. When the user is less involved in the system or when privacy should be enforced, lower-fidelity communication may be most appropriate.

[0043] According to an embodiment of the present disclosure, a video communication system may determine a fidelity based on a selection of multiple different inputs. As an example and not by way of limitation, these inputs may include, but are not limited to, a date and time that may be compared to a user editable schedule, a user's digital calendar (e.g., when the user has a meeting or appointment, fidelity may be limited), user commands, and detected user participation.

[0044] Figure 2An example comparison between high-fidelity video and low-fidelity video is shown. High-fidelity video 210 may be visually clearer and have audio. Low-fidelity video 220 may be blurry and have no audio. Therefore, the video communication system may have technical advantages of avoiding oversharing and privacy protection because the video communication system may adaptively adjust the fidelity or blurriness of the video stream in the video communication based on the connectivity of the corresponding participants.

[0045] According to an embodiment of the present disclosure, the video communication system may determine the corresponding participation level of the participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's position, the participant's body direction, the participant's head direction, the participant's gesture, or the participant's voice for each video stream in the video stream. Therefore, generating a merged video stream may also be based on the corresponding participation level of the participant associated with each video stream in the video stream. The video communication system may automatically determine the user's participation from different inputs. For example, these inputs may include the presence and position of a person, the body and head direction of a person, the gesture of a person, voice detection, the direction of voice arrival, and the like. Detecting the level of participation from inputs including presence and position, body and head direction, gesture, voice, the direction of voice arrival, and the like may be an effective solution for solving the technical challenge of effectively detecting the level of participation of a participant, because these inputs provide comprehensive information that can be used to classify each video stream as the correct level of participation.

[0046] Figure 3 An example flow chart 300 for determining the level of participation at a particular device is shown. The level of participation can be simplified into three categories: full participation, partial participation, and no participation. The example flow chart 300 begins at step 310. At step 320, the video communication system may determine whether anyone has participated in the previous N 1 If not, then at step 330, the video communication system may return "not engaged". If yes, then at step 340, the video communication system may determine whether anyone has been seen in the previous N seconds. 2 seconds. If yes, then at step 350, the video communication system may return to "fully engaged". If no, then at step 360, the video communication system may determine whether the user has been engaged in the previous N seconds. 3 seconds. If yes, then at step 350, the video communication system may return "full participation". If no, then at step 370, the video communication system may return "partial participation". Note that N 1 、N 2 and N 3 Can be constants that determine how long after a certain activity the video communication system reports a higher level of engagement. For example, they can all be set to 60 seconds so that any presence, voice, or attention can affect the output for the next minute.

[0047] According to embodiments of the present disclosure, the fidelity of video and audio sent from the device may be selected based on the level of participation. As an example and not by way of limitation, for "full participation", the video communication system may transmit high fidelity video and audio (e.g., no blur). As another example and not by way of limitation, for "partial participation", the video communication system may transmit low fidelity video and audio (e.g., partial blur). As another example and not by way of limitation, for "no participation", the video communication system may not transmit audio or video.

[0048] According to an embodiment of the present disclosure, a video communication system can merge video streams into a unified scene. The video communication system can be used in groups, where the group size is 2 or larger. When multiple video streams are presented on the same display, the video streams can be automatically adjusted so that the entire display appears as a unified scene rather than a collection of video streams. In other words, the participants can appear to occupy the same space on the display rather than a collection of windows. These adjustments may include, but are not limited to, brightness, color, viewing angle, position (e.g., aligning the head) and blurring of edges. In addition, the relative frame size of each video stream can be adjusted according to the object and activity level of the video stream. For example, a video stream with three people can be allocated a larger proportion of the screen than a video stream with one person. Therefore, the group video call experience can be greatly improved.

[0049] According to an embodiment of the present disclosure, the video communication system may determine, for each video stream in the video stream, a corresponding adjustment of the viewing angle associated with the corresponding video stream so that the object is centered within the frame of the corresponding video stream. Therefore, generating the merged video stream as a unified scene may also be based on the corresponding adjustment of the viewing angle associated with each video stream in the video stream. In an optional embodiment, the video communication system may use depth information (e.g., from a 3D / depth sensor or other method) to de-skew the frames of the video stream. The video communication system may transform the video stream so that the camera appears to be located in the center of the display.

[0050] FIG. 4A to FIG. 4B An example viewing angle adjustment is shown. Figure 4A In FIG. 4 , the video camera 410 may be mounted on top of the display / TV 420. After the viewing angle is adjusted, the real camera field of view 430 may become the apparent field of view 440. Figure 4BIn the example of FIG. 4 , the original images 450-460 with high angles can become adjusted images 470-480 with right angles after the viewing angle adjustment. More information about assisting users subject to privacy settings can be found in U.S. Patent Application No. 16 / 741,630, filed on January 13, 2020, which is incorporated herein by reference. More information about viewing angle adjustment can be found in U.S. Patent Application No. 17 / 666,027, filed on February 7, 2022, which is incorporated herein by reference.

[0051] According to an embodiment of the present disclosure, a video communication system may merge video streams into a unified scene based on a matching scale. The video communication system may scale the video streams so that each person appears to be at a consistent distance from the camera. For simplicity, we introduce the term "vertical distance at object (VDAS)" as a matching scale. VDAS may indicate the total height of a video frame in meters at the depth of an object (person) within the video frame. Using the matching scale, the video communication system may scale each image so that the true distance of each pixel (at the depth of the object) may be the same value for all video streams. This may give the impression that objects are at the same distance from the camera in a unified scene.

[0052] Figure 5 An example process 500 for calculating a scaling factor for each displayed video stream is shown. At step 510, a video communication system may collect images for each displayed stream. At step 520, the video communication system may estimate measurements within the images. At step 530, the video communication system may calculate a VDAS. At step 540, the video communication system may calculate scaling factors for each image to achieve a target VDAS. Subsequent images from each stream may then be scaled according to these scaling factors. Calculating scaling factors based on a vertical distance at an object (VDAS) may be an effective solution for addressing the technical challenge of accurately calculating scaling factors for each video stream, because the resulting scaling factors may enable the video communication system to scale each image so that the true per-pixel distance (at the depth of the object) may be the same value for all video streams, thereby giving the impression that objects in a unified scene are at the same distance from the camera, wherein the vertical distance at the object indicates the total height of the video frame in meters at the depth of the object (person) within the video frame.

[0053] To determine VDAS, the video communication system may first estimate measurements within the image. For each image, a true measurement (in meters) may be estimated at the depth of the object (i.e., a person). If 3D information is available, we can select two points within the object and calculate the straight-line distance between these points in 3D space. If 3D information is not available, we can assume the length of something visible within the image.

[0054] Figure 6 Example assumptions for the length of visible content are shown. For images 610a-610b, we assume the head width is X meters. While head sizes vary from person to person, they can provide a rough sense of scale from the images. For images 620a-620b, we assume the interpupillary distance (IPD) is X meters. The standard deviation of a person's IPD may be only about 6%, making it a reasonably consistent measure. For image 630, we assume the height of the furniture is X meters. Objects may often be seated, and seat heights may be relatively standardized. Therefore, we can assume that the seats for objects are of a standard size. Note that other methods (including combinations of methods) may be used to estimate measurements in the images.

[0055] For the estimated measurements within the image, the video communication system can then calculate the number of pixels occupied by the same length. The video communication system can then calculate the VDAS for each image as:

[0056]

[0057] Where h is the image height in pixels, L is the length measured in meters, and l is the length measured in pixels. Note that the per-pixel distance is L / l.

[0058] Fig. 7A An example measurement of vertical distance at an object is shown. As can be seen, VDAS (in meters) 710 can be determined based on the image height h 720 in pixels, the measured length L 730 in meters, and the measured length l 740 in pixels.

[0059] Figure 7B An example visualization of the vertical distance at an object is shown. Within the field of view 750 of the camera 770 , the VDAS (in meters) 710 may characterize the depth of the object 760 .

[0060] The video communication system may also calculate a scaling factor for each stream. All streams may be scaled up so that they have the same true per-pixel distance (at the depth of the object). There may be a VDAS for each stream. The video communication system may select the lowest VDAS value from all streams, i.e., the VDAS min The scaling factor for each stream can be calculated as

[0061]

[0062] All future images on each stream can then be multiplied by the stream's scaling factor to achieve a similar sense of scale between images. After scaling, all images can have the same per-pixel distance.

[0063] Figure 8Example scaling of images is shown. Images 810a-830a are associated with the same video communication and are before scaling. After scaling, image 810a may become image 810b with a scaling factor of 1.40. Image 820a may become image 820b with a scaling factor of 1.47. Image 830a may become image 830b with a scaling factor of 1.00.

[0064] According to an embodiment of the present disclosure, the video communication system can additionally merge the video streams into a unified scene based on cropping and positioning. Due to potential differences in size and scaling, the video streams may have different sizes. Therefore, the video communication system can determine the crop height, width, and position. Note that the crop heights may need to be equal. Since the per-pixel distance is known during the calculation of the scaling factor, the video communication system can select a target output VDAS in meters and then calculate the crop height in pixels. This can be the same for all images.

[0065] According to an embodiment of the present disclosure, the video communication system may select the target VDAS to be as close to the ideal value (e.g., 1.5 meters) as possible. The target value may be adjusted based on the activity level. For example, in a highly active situation, the target value may be a larger value. The target value may be limited by the size of each image. For example, if the ideal value is 1.5 meters and one of the images has a VDAS of 1.3 meters, the target value may be changed to 1.3 meters. The crop height in pixels may be the target VDAS divided by the distance per pixel. The default crop width for each stream may be the total display width divided by the number of streams displayed. However, the crop width of each stream may be adjusted based on the content within each stream. For each stream, the horizontal position of the crop may be selected to be centered on the person in the view. For each stream, the vertical position of the crop may be selected to achieve consistency in the position of the ground under the person. The ground under each person may not be visible, so a suitable alternative may be to select the vertical position of the crop so that the seat height is consistent.

[0066] According to an embodiment of the present disclosure, cropping may include scaling, such as zooming in or out, and then cropping. Fig. 9 An example cropping of images is shown. Images 910a-930a are associated with the same video communication and are before cropping. The video communication system may determine cropping 912 and cropping vertical position 914 for image 910a, cropping 922 and cropping vertical position 924 for image 920a, and cropping 932 and cropping vertical position 934 for image 930a. After cropping, images 910a, 920a, and 930a may become images 910b, 920b, and 930b, respectively. As can be seen, images 910b-930b may be at the same height as the seats of the three participants of the video communication.

[0067] According to an embodiment of the present disclosure, the video communication system may determine, for each of the video streams, a corresponding adjustment of the color and lighting associated with the corresponding video stream based on the consistency of the visual characteristics across the video streams. Thus, generating a merged video stream may also be based on the corresponding adjustment of the color and lighting associated with each of the video streams. As an example and not by way of limitation, the video communication system may use the viewer's room tone and exposure values ​​to adjust the white balance, brightness, saturation, and other image characteristics to be consistent between camera streams.

[0068] Fig.10 An example block diagram 1000 for adjusting color and lighting is shown. Although the example block diagram 1000 is based on merging two video streams, the principle can be extended to merging any suitable number of video streams. In other words, the pipeline can be run periodically (e.g., once per minute), or when a new user joins, and the rest of the frames from that minute can be processed by the same reference value from the pipeline output.

[0069] At steps 1005a / b, the video communication system may access frames from video stream 1 and video stream 2, respectively.

[0070] In step 1010a / b, the video communication system may apply a light source detector to video stream 1 and video stream 2, respectively. The light source detector may be used to detect the position and relative intensity of a light source in an image.

[0071] At step 1015a / b, the video communication system may apply the light distributor to video stream 1 and video stream 2, respectively. Once the light source is predicted, the video communication system may use this information to evenly redistribute the light across the image.

[0072] In step 1020a / b, the video communication system may obtain the brightness of video stream 1 and video stream 2 respectively.

[0073] In step 1025, the video communication system may select a brightness reference. Specifically, the video communication system may obtain brightness from frames of different video streams (e.g., video stream 1 and video stream 2), average the brightness, or select 1 as the brightness reference. This step may also take into account outdoor lighting using the user's local time, latitude, longitude, and weather conditions. If the outdoor lighting conditions are similar across different users after evaluation, an average reference may be better.

[0074] In step 1030a / b, the video communication system may adjust the brightness of video stream 1 and video stream 2, respectively.

[0075] In step 1035a / b, the video communication system may obtain the next image characteristics of video stream 1 and video stream 2, respectively.

[0076] At step 1040, the video communication system may select a characteristic reference.

[0077] In step 1045a / b, the video communication system may adjust characteristics of video stream 1 and video stream 2, respectively.

[0078] The video communication system may repeat the process including steps 1035-1045 for all desired characteristics. As an example and not by way of limitation, such characteristics may include, but are not limited to, brightness, highlight brightness, shadow brightness, contrast, saturation, color temperature, etc.

[0079] At step 1050a / b, the video communication system generates outputs of video stream 1 and video stream 2, respectively.

[0080] According to an embodiment of the present disclosure, the output may include the modified frame and the reference value.

[0081] Fig.11 Example frames with color and lighting adjustments are shown. Images 1110a-1130a are associated with the same video communication and are before adjusting color and lighting. The video communication system may adjust image 1110a to image 1110b based on adjustments of temperature +30, exposure +11, highlights -30, and shadows +60. The video communication system may adjust image 1120a to image 1120b based on adjustments of temperature -30 and highlights -25. The video communication system may adjust image 1130a to image 1130b based on adjustments of brightness +8, saturation +20, and shadows +50.

[0082] According to an embodiment of the present disclosure, a video communication system can identify a boundary between any two video streams. Thus, generating a merged video stream may include blurring or blending the identified boundary. The boundary between the video streams may be made less obvious by blurring or blending the streams.

[0083] Fig.12 An example block diagram 1200 for implementing image unification is shown. This example block diagram 1200 may indicate how scene unification may be applied. Note that the transformations may not be calculated for each set of images. They may only need to be calculated at the beginning or when there are significant changes.

[0084] At time t 1 , the video communication system may perform the following steps. At step 1210, the video communication system may receive a first set of images. Specifically, the video communication system may receive a frame from each video being displayed. At step 1212, the video communication system may apply a transformation to each image. At step 1214, the video communication system may generate an output image. At step 1216, the video communication system may determine a uniform output characteristic.

[0085] According to an embodiment of the present disclosure, the video communication system may select them to be visually appealing while also minimizing the total amount of change. For example, if all input images have warm tones, the video communication system may select warm tones as the desired output. The output characteristics may include parameters for specifying color, lighting, viewing angle, human scale, etc. At step 1218, the video communication system may calculate the required transformation for each image.

[0086] According to an embodiment of the present disclosure, the video communication system can determine the required transformation so that the image has the desired output characteristics. For example, the video communication system can increase the brightness, change the color temperature, vertically shift, etc.

[0087] At time t 2 , the video communication system may perform the following steps. At step 1220, the video communication system may receive a second set of images. At step 1222, the video communication system may apply a transformation to each image. In other words, each image may have its own transformation applied. At step 1224, the video communication system may generate an output image that is ready to be arranged or tiled for display.

[0088] At time t 3 At time t, the video communication system may execute 2 The steps are similar to the steps at .

[0089] Fig.13 An example blending of borders is shown. Images 1310-1330 are associated with the same video communication and are before their borders are blended. The video communication system may blend their borders to obtain a unified display 1340.

[0090] After the video communication system scales, filters, transforms, and positions the video streams as described above, the video communication system may have another technical advantage of presenting multiple video streams in the video communication as a unified scene.

[0091] According to an embodiment of the present disclosure, the video communication system may determine, for each video stream in the video stream, a corresponding adjustment of the spatial allocation associated with the corresponding video stream based on one or more of the following items: a count of participants present in the corresponding video stream, a degree of participation of participants in the corresponding video stream, or an activity level of participants in the corresponding video stream. Therefore, generating a merged video stream may also be based on a corresponding adjustment of the spatial allocation associated with each video stream in the video stream. In other words, the video communication system may further improve the presentation of multiple video streams by automatically adjusting the size of the video streams. Depending on the content within each stream, it may be advantageous to allocate more space to certain streams. Relevant information may be automatically detected by video analysis (including human and gesture detection).

[0092] Figures 14A-14C An example situation is shown in which the stream size may be adjusted to obtain a better group call experience.

[0093] Fig.14A An example case of adjusting the stream size based on the number of people is shown. The more people there are in the video stream, the more area can be allocated for the video stream. The original video streams can be video stream 1410a and video stream 1420a. Since there are more people in video stream 1410a than in video stream 1420a, the video communication system can adjust their stream sizes to generate video stream 1410b and video stream 1420b.

[0094] Fig. 14B An example case of adjusting stream size based on participation is shown. When a participant associated with a video stream is not participating in or paying attention to the video communication, the video communication system can allocate less area for the video stream. The original video streams can be video stream 1430a and video stream 1440a. Since the person in video stream 1430a is not paying attention to the video communication compared to the person in video stream 1440a, the video communication system can adjust their stream sizes to produce video stream 1430b and video stream 1440b.

[0095] Fig. 14C An example case of adjusting stream size based on activity level is shown. The video communication system can allocate more space to a video stream when a participant associated with the video stream (e.g., a child showing dance moves) is moving. The original video streams may be video stream 1450a and video stream 1460a. Since the person in video stream 1450a is dancing and the person in video stream 1460a is sitting still, the video communication system may adjust their stream sizes to produce video stream 1450b and video stream 1460b. Since the video communication system can allocate appropriate space to each video stream based on participant count, participation, and activity level, the video communication system may have another technical advantage of automatically adjusting relative video stream sizes according to content.

[0096] According to an embodiment of the present disclosure, a video communication system can enable users to control their video streams. Gestures can be used to issue commands to the video communication system. These gestures can be recognized by analyzing the output of one or more sensors (such as a camera, a lidar, or an event camera). The analysis can be calculated locally on the device of the gesture performer or remotely on a server. Gestures can be used in combination with voice commands.

[0097] According to an embodiment of the present disclosure, a user may use a gesture to turn off a privacy filter. For example, the user may raise his hand. A user interface (UI) may appear to indicate gesture recognition, such as "hold your hand for one second". The UI may also provide a visual prompt, such as a countdown or progress bar. After gesture recognition, the video communication system may turn off the privacy filter.

[0098] According to an embodiment of the present disclosure, a user may use a gesture to turn on a privacy filter. For example, the user may raise a hand and start waving. The user may additionally say some words, such as "bye," "see you next time," "goodbye," "good night," etc. After a few seconds, the video communication system may turn on the privacy filter.

[0099] According to an embodiment of the present disclosure, a user can use gestures to highlight the video stream from their camera on the screen of other users. For example, a user can perform a gesture of pushing a wall away, which can trigger the video communication system to make the video stream from the user's camera full screen.

[0100] According to an embodiment of the present disclosure, a video communication system can provide interactive video overlay control to a user. The video overlay can be controlled by the user's movement (e.g., hand placement, limb placement, body placement, gestures, expressions, head direction, eye direction, etc.).

[0101] Figures 15A-15C An example interactive video overlay control is shown.

[0102] Fig.15A An example gesture to trigger an overlay menu is shown. A user 1510 can use a gesture such as a stationary open palm to trigger the overlay menu to appear. The overlay menu can include three menu items, such as "Private Mode" 1520, "Adjust Volume" 1530, and "Off" 1540. The location of the menu items can depend on the user's location and gestures. They can be selected to be easily accessible by the user. The overlay menu can also expand the frame size of the user's video to create more space.

[0103] Fig. 15B An example selection of a menu item is shown. A menu item may be selected by a user moving their hand to the overlay. Fig. 15B , select “Adjust Volume” 1530. Fig. 15C An example slider in a video overlay is shown. User 1510 can select a value for adjusting the volume by selecting a position along slider 1550 using the user's 1510 hand position to simulate slider 1550.

[0104] According to an embodiment of the present disclosure, the video communication system may additionally transmit information about the video communication system to the user. Such information may include the type of data sent from the system (e.g., audio, video, blur level), the user from which the video can be received, the user currently receiving the video from, the user layout on the display, etc. The video communication system can provide a clear indication of whether high-fidelity video or audio is being shared by using indicator LEDs. Possible form factors of LED indicators may include, but are not limited to, strips, rings, or other shapes on the front of the camera, patterns that illuminate the wall on the back of the camera or client device. Therefore, these indicator LEDs can enable users to quickly view to assess the degree of sharing, thereby preventing unexpected sharing when using permanently connected video communications.

[0105] According to an embodiment of the present disclosure, the video communication system can also utilize a remote microphone. The remote microphone can be placed in a position more suitable for hearing the user's voice, such as on a coffee table. The remote microphone can communicate with the video communication system via a fixed cable or wirelessly. The remote controller can include an LED indicator, and the remote microphone can be built into the remote controller. The remote microphone can use a rechargeable battery and / or a solar panel.

[0106] According to an embodiment of the present disclosure, a video communication system may meet the following criteria. One criterion may be that the initiator of the video stream has access to a wide field of view. The region of interest within the shared space may change depending on the location of the object. Therefore, the camera system of the initiator of the video stream may be able to access a wide field of view to capture a variety of ranges of interest. Another criterion may be that the recipient of the video stream can display a narrow field of view. It may be desirable for the recipient's display to show the region of interest in an attractive scale and detail. Streaming the entire wide-angle camera view may generally be a poor choice because the range of interest may constitute a small portion of the total video stream. The video communication system may utilize various methods to meet the above criteria, which may include but are not limited to wide-angle lenses, automatic cropping, wide-angle lens correction, motorized cameras, optical zoom, and multiple cameras that are typically used to capture a wide field of view from a single lens. With automatic cropping, the region of interest can be cropped within a larger field of view. The video communication system may automatically determine the cropping region from a series of inputs, such as the location of the object and its activity. The video communication system may use wide-angle lens correction to correct the distortion introduced by the wide-angle lens. Wide angle lens correction can be performed only on a cropped portion of the video, which can provide better results than if the correction was performed on the entire image before cropping. For motorized cameras, the camera can be mounted on a gimbal or other rotating element that allows the camera to point to an area of ​​interest. This can enable the camera to access a larger field of view without having to change to a wide angle lens. Optical zoom can allow the field of view to be narrowed without losing resolution. Multiple cameras can be combined to provide the benefits of different cameras. For example, a wide angle lens can be used to maintain an overall view, while a motorized telephoto lens can provide a high-quality close-up view.

[0107] Fig.16 An example application of the video communication system is shown. The video communication system described herein can be applied to any device that supports video communication. Such a device can have a range of form factors. Options can include, but are not limited to, software on a mobile device 1610, software on a television unit 1620 optionally with an attached camera unit, software on a TV connection platform 1630 optionally with an attached camera unit, an all-in-one dedicated device connected to a TV 1640, a dedicated device with an attached camera unit 1650, etc.

[0108] According to an embodiment of the present disclosure, a video communication system can be used for upcycling. Upcycling is the process of converting an undesirable product into a new product with a desired use.

[0109] Fig.17An example process 1700 of upscaling is shown. The video communication system may consist of software on a mobile device that is semi-permanently mounted around a television. At step 1710, the software may be loaded onto the mobile device. The camera, microphone, processor, display, speaker, network, and other capabilities of the mobile device may be utilized. At step 1720, the mobile device may be placed in a bracket. At step 1730, the mobile device may be mounted around the TV. The bracket may be used to support the desired installation of the mobile device, for example by providing power. The bracket may also provide access to a display interface such as HDMI that will allow the mobile device to output video and audio to the TV. Wireless display transmission may be used to send video and audio output to the TV.

[0110] Fig.18 A flowchart of a method 1800 for improving the presentation of multiple video streams in video communication according to an embodiment of the present disclosure is shown. The method 1800 may be performed using one or more processing devices (e.g., electronic device 100), which may include hardware (e.g., a general purpose processor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a system on a chip (SoC), a microcontroller, a field programmable gate array (FPGA), a central processing unit (CPU), an application processor (AP), a visual processing unit (VPU), a neural processing unit (NPU), a neural decision processor (NDP), or any other processing device that may be suitable for processing 2D and 3D image data, software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or some combination thereof.

[0111] The method 1800 may be started at step 1810 using one or more processing devices (e.g., electronic device 100). For example, according to an embodiment of the present disclosure, the electronic device 100 may maintain video communication between two or more client devices, wherein each client device is associated with a corresponding video stream in the video communication, and wherein each video stream is associated with a corresponding matching scale based on a height of a frame of the video stream and a depth measurement of an object within the frame.

[0112] Then, the method 1800 may continue at step 1820 using one or more processing devices (e.g., the electronic device 100). For example, the electronic device 100 may determine, for each of the video streams, a respective scaling factor and a respective crop, wherein the respective scaling factor is calculated based on a matching scale associated with the respective video stream and a scale target matching scale determined from matching scales associated with video streams associated with two or more client devices, and wherein the respective crop is determined based on consistency of a position of an object within frames across the video streams associated with the two or more client devices.

[0113] Then, the method 1800 may continue at step 1830 using one or more processing devices (e.g., the electronic device 100). For example, according to an embodiment of the present disclosure, the electronic device 100 may determine, for each video stream, a corresponding fidelity based on one or more of a date and time associated with the corresponding video stream, a calendar of a participant associated with the corresponding video stream, a command from a participant associated with the corresponding video stream, or a degree of participation of a participant associated with the video stream (the corresponding degree of participation of a participant associated with the video stream is based on a location of the participant, a body orientation of the participant, a head orientation of the participant, a gesture of the participant, or a voice of the participant), determine a corresponding adjustment of the perspective associated with the corresponding video stream to center an object within a frame of the corresponding video stream, determine a corresponding adjustment of the color and lighting associated with the corresponding video stream based on consistency of visual characteristics across the video streams, and determine a corresponding adjustment of the spatial allocation associated with the corresponding video stream based on one or more of a count of participants present in the corresponding video stream, a degree of participation of participants in the corresponding video stream, or an activity level of participants in the corresponding video stream.

[0114] Then, the method 1800 may continue at block 1840 using one or more processing devices (eg, the electronic device 100). For example, according to an embodiment of the present disclosure, the electronic device 100 may identify a boundary between any two video streams.

[0115] Then, the method 1800 may continue at step 1850 using one or more processing devices (e.g., the electronic device 100). For example, according to an embodiment of the present disclosure, the electronic device 100 may generate a merged video stream from each of the video streams for video communication based on the corresponding scaling factor, cropping, fidelity, participation of the participant, adjustment of the viewing angle, and adjustment of color and lighting associated with each video stream, wherein generating the merged video stream includes blurring or blending the identified boundaries.

[0116] Then, the method 1800 may continue at block 1860 using one or more processing devices (e.g., electronic device 100). For example, according to an embodiment of the present disclosure, the electronic device 100 may send instructions to one or more of the client devices for presenting the merged video stream. Where appropriate, certain embodiments may repeat Fig.18 Although the present disclosure will Fig.18 The specific steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates Fig.18 Any suitable steps of the method may occur in any suitable order.

[0117] Furthermore, although the present disclosure describes and illustrates Fig.18The present disclosure contemplates any suitable method for improving the presentation of multiple video streams in video communications including any suitable steps, which, where appropriate, may include Fig.18 In addition, although the present disclosure describes and illustrates the implementation of Fig.18 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Fig.18 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0118] System and method

[0119] Fig.19 An exemplary computer system 1900 is shown that may be used to perform improved presentation of multiple video streams in video communications in accordance with embodiments of the present disclosure.

[0120] According to an embodiment of the present disclosure, one or more computer systems 1900 execute one or more steps of one or more methods described or illustrated herein.

[0121] According to an embodiment of the present disclosure, one or more computer systems 1900 provide the functionality described or illustrated herein.

[0122] According to an embodiment of the present disclosure, software running on one or more computer systems 1900 performs one or more steps of one or more methods described or shown herein or provides the functions described or shown herein. Specific embodiments include one or more parts of one or more computer systems 1900. In this document, references to computer systems may include computing devices and vice versa, where appropriate. In addition, references to computer systems may include one or more computer systems, where appropriate.

[0123] The present disclosure contemplates any suitable number of computer systems 1900. The present disclosure contemplates computer systems 1900 taking any suitable physical form. As an example and not by way of limitation, computer system 1900 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on a module (COM) or a system on a module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these devices. Where appropriate, computer system 1900 may include one or more computer systems 1900; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks.

[0124] Where appropriate, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. As an example and not by way of limitation, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 1900 may perform one or more steps of one or more methods described or illustrated herein at different times or at different locations.

[0125] According to an embodiment of the present disclosure, computer system 1900 includes a processor 1902, a memory 1904, a storage device 1906, an input / output (I / O) interface 1908, a communication interface 1910, and a bus 1912. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0126] According to an embodiment of the present disclosure, the processor 1902 includes hardware for executing instructions (such as those constituting a computer program). As an example and not by way of limitation, to execute instructions, the processor 1902 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1904, or a storage device 1906; decode and execute them; and then write one or more results to an internal register, an internal cache, a memory 1904, or a storage device 1906.

[0127] According to embodiments of the present disclosure, the processor 1902 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 1902 that includes any suitable number of any suitable internal caches. As an example, and not by way of limitation, the processor 1902 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions in the instruction cache may be copies of instructions in the memory 1904 or storage device 1906, and the instruction cache may speed up the retrieval of those instructions by the processor 1902.

[0128] The data in the data cache may be a copy of data in the memory 1904 or storage device 1906 for operation by instructions executed at the processor 1902; the results of previous instructions executed at the processor 1902 for access by subsequent instructions executed at the processor 1902 or for writing to the memory 1904 or storage device 1906; or other suitable data. The data cache may speed up the read or write operation of the processor 1902. The TLB may speed up the virtual address translation of the processor 1902.

[0129] According to embodiments of the present disclosure, processor 1902 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates processor 1902 including any suitable number of any suitable internal registers. Where appropriate, processor 1902 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1902. Although the present disclosure describes and illustrates a particular processor, the present disclosure contemplates any suitable processor.

[0130] According to an embodiment of the present disclosure, the memory 1904 includes a main memory for storing instructions for execution by the processor 1902 or data for operation by the processor 1902. As an example and not by way of limitation, the computer system 1900 can load instructions from the storage device 1906 or another source (such as, for example, another computer system 1900) to the memory 1904. Then, the processor 1902 can load the instructions from the memory 1904 to an internal register or an internal cache. In order to execute the instructions, the processor 1902 can retrieve the instructions from the internal register or the internal cache and decode them. During or after the execution of the instructions, the processor 1902 can write one or more results (which can be intermediate or final results) to the internal register or the internal cache. Then, the processor 1902 can write one or more of those results to the memory 1904.

[0131] According to an embodiment of the present disclosure, the processor 1902 only executes instructions in one or more internal registers or internal caches or in the memory 1904 (different from the storage device 1906 or other places), and only operates on data in one or more internal registers or internal caches or in the memory 1904 (different from the storage device 1906 or other places).

[0132] One or more memory buses (each of which may include an address bus and a data bus) may couple processor 1902 to memory 1904. As described below, bus 1912 may include one or more memory buses.

[0133] According to an embodiment of the present disclosure, one or more memory management units (MMUs) are located between the processor 1902 and the memory 1904 , and facilitate access to the memory 1904 requested by the processor 1902 .

[0134] According to an embodiment of the present disclosure, the memory 1904 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 1904 may include one or more memory devices 1904. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0135] According to an embodiment of the present disclosure, storage device 1906 includes a large-capacity storage device for data or instructions. As an example and not by way of limitation, storage device 1906 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these devices. Where appropriate, storage device 1906 may include removable or non-removable (or fixed) media. Where appropriate, storage device 1906 may be inside or outside of computer system 1900.

[0136] According to an embodiment of the present disclosure, the storage device 1906 is a non-volatile solid-state memory.

[0137] According to an embodiment of the present disclosure, the storage device 1906 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory or a combination of two or more of these devices. The present disclosure contemplates a mass storage device 1906 in any suitable physical form. Where appropriate, the storage device 1906 may include one or more storage control units that facilitate communication between the processor 1902 and the storage device 1906. Where appropriate, the storage device 1906 may include one or more storage devices 1906. Although the present disclosure describes and illustrates specific storage devices, the present disclosure contemplates any suitable storage device.

[0138] According to an embodiment of the present disclosure, the I / O interface 1908 includes hardware, software, or both of one or more interfaces for communication between the computer system 1900 and one or more I / O devices. Where appropriate, the computer system 1900 may include one or more of these I / O devices. One or more of these I / O devices can implement communication between a person and the computer system 1900. As an example and not by way of limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a static camera, a stylus, a tablet, a touch screen, a trackball, a video camera, other suitable I / O devices, or a combination of two or more of these devices. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1906 for them. Where appropriate, the I / O interface 1908 may include one or more devices or software drivers that enable the processor 1902 to drive one or more of these I / O devices. Where appropriate, the I / O interface 1908 may include one or more I / O interfaces 1906. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0139] According to an embodiment of the present disclosure, the communication interface 1910 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 1900 and one or more other computer systems 1900 or one or more networks. As an example and not by way of limitation, the communication interface 1910 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other cable-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1910 for it.

[0140] As an example and not by way of limitation, the computer system 1900 may communicate with one or more parts of a self-organizing network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these networks. One or more parts of one or more of these networks may be wired or wireless. As an example, the computer system 1900 may communicate with a wireless PAN (WPAN) (such as, for example, a Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular phone network (such as, for example, a global system for mobile communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 1900 may include any suitable communication interface 1910 for any of these networks. Where appropriate, the communication interface 1910 may include one or more communication interfaces 1910. Although the present disclosure describes and illustrates a specific communication interface, the present disclosure contemplates any suitable communication interface.

[0141] According to an embodiment of the present disclosure, bus 1912 includes hardware, software, or both that couple components of computer system 1900 to each other. As an example and not by way of limitation, bus 1912 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or other suitable bus or a combination of two or more of these buses. Where appropriate, bus 1912 may include one or more buses 1912. Although the present disclosure describes and illustrates a specific bus, the present disclosure contemplates any suitable bus or interconnect.

[0142] One embodiment provides a method comprising: maintaining, by an electronic device, a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication;

[0143] One embodiment provides a method comprising: an electronic device automatically adjusting the corresponding fidelity for each video stream in the video stream based on one or more of a date and time associated with the corresponding video stream, a calendar of a participant associated with the corresponding video stream, a command from a participant associated with the corresponding video stream, or a level of participation of a participant associated with the corresponding video stream.

[0144] One embodiment provides a method comprising: an electronic device determining, for each video stream in the video stream, a corresponding level of participation of a participant associated with the corresponding video stream based on one or more of the participant's presence, the participant's position, the participant's body direction, the participant's head direction, the participant's gestures, or the participant's voice.

[0145] Miscellaneous

[0146] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Furthermore, "and" is both jointly and severally, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated otherwise by context.

[0147] As used herein, "automatically" and its derivatives mean "without human intervention" unless expressly stated otherwise or indicated otherwise by context.

[0148] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Embodiments according to the present invention are specifically disclosed in the attached claims for methods, storage media, systems, and computer program products, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or reverse references in the attached claims are selected only for formal reasons. However, any subject matter generated by an intentional reference to any previous claim (especially multiple dependencies) may also be claimed, so that any combination of claims and their features is disclosed and may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. In addition, any embodiments and features described or depicted herein may be claimed in separate claims and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the attached claims.

[0149] The scope of the present disclosure covers all changes, substitutions, variations, alterations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and depicts the corresponding embodiments herein as including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, the attached claims refer to the device or system or the component of the device or system being suitable for, arranged to, capable of, configured to, enabled to, operable or functioning to perform a specific function, covering the device, system, component whether it or the specific function is activated, turned on or unlocked, as long as the device, system or component is so suitable for, arranged, capable of, configured, enabled, operable or functioning. In addition, although the present disclosure describes or illustrates a specific embodiment as providing a specific advantage, a specific embodiment may not provide these advantages, provide some or all of these advantages.

Claims

1. A method, include: The following operations are performed by the electronic device: maintaining a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication, and wherein each video stream is associated with a respective matching metric based on a height of a frame of the each video stream and a depth measurement of an object within the frame; determining, for each of the video streams, a respective scaling factor and a respective crop, wherein the respective scaling factor is calculated based on the matching metric associated with the respective video stream and a target matching metric determined from matching metric associated with the video streams associated with the two or more client devices, and wherein the respective crop is determined based on consistency of positions of objects within frames across the video streams associated with the two or more client devices; generating a merged video stream from each of the video streams for use in the video communication based on the respective scaling factors and the respective crops of each video stream; and Instructions are sent to one or more of the client devices to present the merged video stream.

2. The method according to claim 1, further comprising: include: For each of the video streams, a respective fidelity is determined based on one or more of a date and time associated with the respective video stream, a calendar of a participant associated with the respective video stream, a command from the participant associated with the respective video stream, or a level of engagement of the participant associated with the video stream, wherein generating the merged video stream is also based on the respective fidelity associated with each of the video streams.

3. The method according to claim 1 or claim 2, further comprising: include: For each of the video streams, a corresponding level of participation of the participant is determined based on one or more of the presence of the participant associated with the corresponding video stream, the position of the participant, the body orientation of the participant, the head orientation of the participant, the gesture of the participant, or the voice of the participant, wherein generation of the merged video stream is also based on the corresponding level of participation of the participant associated with each of the video streams.

4. The method according to any one of the preceding claims, further comprising: include: For each of the video streams, determining a corresponding adjustment of the perspective associated with the corresponding video stream, the corresponding adjustment being used to center the object within the frame of the corresponding video stream, wherein generating the merged video stream is also based on the corresponding adjustment of the perspective associated with each of the video streams.

5. The method according to any one of the preceding claims, further comprising: include: For each of the video streams, a respective adjustment of color and lighting associated with the respective video stream is determined based on consistency of visual characteristics across the video streams, wherein generating the merged video stream is also based on the respective adjustment of color and lighting associated with each of the video streams.

6. The method according to any one of the preceding claims, further comprising: include: A boundary between any two of the video streams is identified, wherein generating the merged video stream includes blurring or blending the identified boundary.

7. The method according to any one of the preceding claims, further comprising: include: For each of the video streams, a corresponding adjustment of the spatial allocation associated with the corresponding video stream is determined based on one or more of the following items: a count of participants present in the corresponding video stream, a degree of engagement of participants in the corresponding video stream, or an activity level of participants in the corresponding video stream, wherein generating the merged video stream is also based on the corresponding adjustment of the spatial allocation associated with each of the video streams.

8. An electronic device, include: one or more displays; one or more non-transitory computer-readable storage media comprising instructions; and One or more processors, coupled to the storage medium, the one or more processors configured to execute the instructions to: maintaining a video communication between two or more client devices, wherein each client device is associated with a respective video stream in the video communication, and wherein each video stream is associated with a respective matching metric based on a height of a frame of the each video stream and a depth measurement of an object within the frame; determining, for each of the video streams, a respective scaling factor and a respective crop, wherein the respective scaling factor is calculated based on the matching metric associated with the respective video stream and a target matching metric determined from matching metric associated with the video streams associated with the two or more client devices, and wherein the respective crop is determined based on consistency of positions of objects within frames across the video streams associated with the two or more client devices; generating a merged video stream from each of the video streams for use in the video communication based on the respective scaling factors and the respective crops of each video stream; and Instructions are sent to one or more of the client devices to present the merged video stream.

9. The electronic device according to claim 8, in, The processor is further configured to execute the instructions to: For each of the video streams, a respective fidelity is determined based on one or more of a date and time associated with the respective video stream, a calendar of a participant associated with the respective video stream, a command from the participant associated with the respective video stream, or a level of engagement of the participant associated with the video stream, wherein generating the merged video stream is also based on the respective fidelity associated with each of the video streams.

10. The electronic device according to claim 8 or claim 9, wherein the processor is further configured to execute the instructions to: for each of the video streams, determining a respective level of engagement of the participant based on one or more of a presence of the participant, a position of the participant, a body orientation of the participant, a head orientation of the participant, a gesture of the participant, or a voice of the participant associated with the respective video stream, in, Generating the merged video stream is also based on the respective participation levels of the participants associated with each of the video streams.

11. An electronic device according to any one of the preceding claims, in, The processor is further configured to execute the instructions to: For each of the video streams, determining a corresponding adjustment of the perspective associated with the corresponding video stream, the corresponding adjustment being used to center the object within the frame of the corresponding video stream, wherein generating the merged video stream is also based on the corresponding adjustment of the perspective associated with each of the video streams.

12. An electronic device according to any one of the preceding claims, in, The processor is further configured to execute the instructions to: For each of the video streams, a respective adjustment of color and lighting associated with the respective video stream is determined based on consistency of visual characteristics across the video streams, wherein generating the merged video stream is also based on the respective adjustment of color and lighting associated with each of the video streams.

13. An electronic device according to any one of the preceding claims, in, The processor is further configured to execute the instructions to: A boundary between any two of the video streams is identified, wherein generating the merged video stream includes blurring or blending the identified boundary.

14. An electronic device according to any one of the preceding claims, in, The processor is further configured to execute the instructions to: For each of the video streams, a corresponding adjustment of the spatial allocation associated with the corresponding video stream is determined based on one or more of the following items: a count of participants present in the corresponding video stream, a degree of engagement of participants in the corresponding video stream, or an activity level of participants in the corresponding video stream, wherein generating the merged video stream is also based on the corresponding adjustment of the spatial allocation associated with each of the video streams.

15. A computer-readable medium comprising at least one instruction, which, when executed, causes at least one processor of a device to perform operations corresponding to the method of any one of claims 1 to 7.

Images