Video conferencing system for hybrid meetings

Abstract

The method includes: a first remote device associated with a first remote attendee receiving a remote video stream from a second remote device containing content associated with a second remote attendee of a video conference; the first remote device receiving a local video stream from a local device associated with a local attendee of a video conference containing instructions for the display order of the first remote attendee, the second remote attendee, and the local attendee; and rendering the remote video stream and the local video stream on the first device based on the display order.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the priority of U.S. Provisional Patent Application No. 63 / 493,307, filed on March 30, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] Embodiments relate to video conferencing systems.

Background Art

[0003] In hybrid meetings (e.g., video conferencing meetings), there may be a mix of attendees. For example, some attendees may be in an office conference room, while other attendees may be remote attendees who participate in the meeting from a remote location such as their home or another office.

Summary of the Invention

[0004] Embodiments relate to a video conferencing system that uses hardware and software configured to simulate local presence by remote attendees. Simulating local presence by remote attendees is sometimes referred to as a technique for implementing a presence enhancement feature. The video conferencing system can be configured to display a visually consistent video feed among segmented human participants, or in other words, among video conferencing attendees at different physical locations (e.g., local and remote attendees).

[0005] In a general embodiment, a device, system, non-temporary computer-readable medium (containing computer-executable program code that can be run on a computer system), and / or method may perform a process by a method that includes: a first remote device associated with a first remote attendee receiving a remote video stream from a second remote device containing content associated with a second remote attendee of a video conference; the first remote device receiving a local video stream from a local device associated with a local attendee of a video conference containing instructions for the display order of the first remote attendee, the second remote attendee, and the local attendee; and rendering the remote video stream and the local video stream on the first device based on the display order.

[0006] In other common embodiments, a device, system, non-temporary computer-readable medium (containing computer-executable program code that can be run on a computer system), and / or method may perform a process by method including generating a user interface (UI) including a first display portion corresponding to a first camera and a second display portion corresponding to a second camera; rendering the UI onto the display of a video conferencing system; receiving a video stream corresponding to a remote attendee of a video conference; and rendering the remote attendee in one of the first or second display portions, wherein rendering includes normalizing the presentation of the remote attendee to the local attendee.

[0007] In other common embodiments, a device, system, non-temporary computer-readable medium (containing computer-executable program code that can be run on a computer system), and / or method may perform a process by method including generating a user interface (UI) including a first display portion corresponding to a first camera and a second display portion corresponding to a second camera; rendering the UI on the display of a video conferencing system; receiving a video stream corresponding to remote attendees of a video conference; and generating a video stream including local attendees, the video stream including instructions on the display order of local attendees to remote attendees.

[0008] Exemplary embodiments will be better understood from the embodiments for carrying out the invention described below herein and the accompanying drawings. Similar elements are represented by the same reference numerals and are shown for illustrative purposes only and are not limiting to the exemplary embodiments. [Brief explanation of the drawing]

[0009] [Figure 1A] A front block diagram of a video conferencing system according to at least one exemplary embodiment is shown. [Figure 1B] A front block diagram of a video conferencing system according to at least one exemplary embodiment is shown. [Figure 1C] A front block diagram of a video conferencing system according to at least one exemplary embodiment is shown. [Figure 2] A top view of a photographic representation of a video conferencing system according to at least one exemplary embodiment is shown. [Figure 3] A front block diagram of a video conferencing system display, according to at least one exemplary embodiment, is shown. [Figure 4] A block diagram of a video conferencing system according to at least one exemplary embodiment is shown. [Figure 5] A rear block diagram of the display is shown, according to at least one exemplary embodiment. [Figure 6A] A block diagram of a multi-camera display arrangement according to at least one exemplary embodiment is shown. [Figure 6B] A block diagram of a multi-camera display arrangement according to at least one exemplary embodiment is shown. [Figure 6C] A block diagram of a multi-camera display arrangement according to at least one exemplary embodiment is shown. [Figure 6D] A block diagram of the display order according to at least one exemplary embodiment is shown. [Figure 6E] A block diagram of the display order according to at least one exemplary embodiment is shown. [Figure 7] This is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. [Figure 8] This is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. [Figure 9] This is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. [Modes for carrying out the invention]

[0010] It should be noted that these drawings are intended to illustrate general characteristics of the methods and / or structures used in specific exemplary embodiments and to supplement the written descriptions provided below. However, these drawings are not to scale and may not accurately reflect the exact structural or performance characteristics of any given embodiment and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various drawings is intended to indicate the existence of similar or identical elements or features.

[0011] Hybrid meetings (e.g., video conference meetings) can have a mix of attendees. For example, some attendees (e.g., local attendees) may be in an office meeting room, while others are remote attendees joining the meeting from home or another office. At least one problem may be that hybrid meetings using some video conferencing systems are not as effective as in-person meetings. For example, in hybrid meetings, the flow of conversation can be awkward and stiff, it can be difficult to read body language, other important social signals such as eye contact and direction of attention may be missed and / or misinterpreted, people may feel distant and find it difficult to form strong social bonds, remote attendees are often ignored and cannot participate fairly in the meeting, and / or collaborative brainstorming tasks are difficult.

[0012] At least one technical solution to this problem is a video conferencing system that uses hardware and software configured to simulate local attendance by remote attendees, sometimes referred to as presence enhancement features. The video conferencing system can be configured to display a visually consistent (e.g., consistent display order) video feed among segmented human participants (e.g., local and remote attendees). For example, local attendees can see remote attendees in a certain order, and remote attendees can see other remote attendees in the same order, and local attendees are displayed as if they were inserted within the same order (see counterclockwise order below). The video conferencing system may be interoperable with existing systems and may run on existing network systems.

[0013] Figure 1A shows a front block diagram of a video conferencing system according to at least one exemplary embodiment. As shown in Figure 1A, the video conferencing system 100 includes displays 105A, 105B, and 105C, and peripheral device strips 110A, 110B, and 110C. Figure 1B shows a front block diagram of a video conferencing system according to at least one exemplary embodiment. As shown in Figure 1B, the video conferencing system 100 includes a display 105D, and peripheral device strips 110A, 110B, and 110C. Figure 1A shows a video conferencing system 100 using multiple displays (e.g., three (3)), and Figure 1B shows a video conferencing system 100 with a single wide display. A wide display may be a display with a wide aspect ratio. For example, a wide display may have an aspect ratio of 16:9.

[0014] As shown in Figures 1A and 1B, remote attendees 115A, 115B, and 115C displayed on displays 105A, 105B, and 105C, and local attendees 120A and 120B, can be located locally, for example, in a conference room. Local attendees 120A and 120B can be seated, for example, at a table (not shown). The video conferencing system 100 can be configured to implement a presence enhancement function when displaying remote attendees 115A, 115B, and 115C on displays 105A, 105B, and 105C. The peripheral device strips 110A, 110B, and 110C can be housings that can hold non-display devices together as a unit. The peripheral device strips 110A, 110B, and 110C can include, for example, cameras and speakers. In some embodiments, peripheral device strips 110A, 110B, 110C may include, for example, three cameras and two speakers. Remote attendees may be associated with a remote device (e.g., a computing device) operating during the video conference, along with a local device operating during the video conference. Thus, the local device may correspond to the video conferencing system 100, and the remote device may be any other computing device operating during the video conference. The remote device may be configured to stream video associated with the remote attendee. The local device may be configured to stream video associated with the local attendee. The video stream may be associated with the video conference.

[0015] For example, remote attendees 115A, 115B, and 115C can be rendered on displays 105A, 105B, and 105C using natural scaling rendering. For example, remote attendees 115A, 115B, and 115C can be rendered on displays 105A, 105B, and 105C using optimized display space allocation. For example, remote attendees 115A, 115B, and 115C can be rendered on displays 105A, 105B, and 105C using a common background. For example, video streams sent to remote attendees 115A, 115B, and 115C can be streamed using spatially consistent mapping. In other words, the display order or rendering position of remote attendees 115A, 115B, and 115C can be consistent across local and remote video conferencing systems.

[0016] In some embodiments, a video conference can include a local video conference system (e.g., video conference system 100) and at least one remote video conference system (e.g., video conference system 150). As shown in FIGS. 1A and 1B, the left-to-right mapping, display order, or rendering position of remote attendees are remote attendee 115A, remote attendee 115B, and remote attendee 115C. Further, the position (or physical location) of local attendees is such that local attendee 120A is between remote attendee 115A and remote attendee 115B, and local attendee 120B is between remote attendee 115B and remote attendee 115C. The attendee mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc. can be from one of many viewpoints. For example, the attendee mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc. can be a circular arrangement clockwise or counterclockwise as seen from above. For example, the attendee mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc. can be seen as a linear arrangement from left to right or right to left from the perspective of a remote attendee. For example, the attendee mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc. can be seen as a linear arrangement from left to right or right to left from the perspective of a local attendee.

[0017] Therefore, the mapping of attendees, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc., can be remote attendee 115A, remote attendee 115B, remote attendee 115C, local attendee 120B, and local attendee 120A (for example, clockwise), or remote attendee 115C, remote attendee 115B, remote attendee 115A, local attendee 120A, and local attendee 120B (for example, counterclockwise). Therefore, the mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, and rendering location of attendees can be, for example, remote attendee 115A, local attendee 120A, remote attendee 115B, local attendee 120B, and remote attendee 115C (e.g., left to right), or remote attendee 115C, local attendee 120B, remote attendee 115B, local attendee 120A, and remote attendee 115A (e.g., right to left). This mapping or display order must be consistent across all video conferencing systems during a video conference. Note that remote attendees may not be rendered on remote video conferencing systems, but other attendees must be rendered according to the above mapping or display order. For example, remote attendees 115A, 115B, and 115C can be rendered on displays 105A, 105B, and 105C using localized audio. For example, remote attendees 115A, 115B, and 115C can be rendered on displays 105A, 105B, and 105C with normalized eye-levels. In some embodiments, the video conferencing system 100 may have fewer than three cameras (e.g., one or two). In this case, remote attendees can be associated with the nearest camera and / or the default camera.

[0018] FIG. 1C shows a front block diagram of a video conferencing system according to at least one exemplary embodiment. As shown in FIG. 1C, the video conferencing system 150 can be used by a remote attendee 115A to participate in a video conference (e.g., a hybrid video conference). The video conferencing system 150 can represent a remote video conferencing system communicatively coupled to the video conferencing system 100. In some embodiments, the video conferencing system 150 can include a display 155, a camera 160, and a speaker(s) 165. The display 155 can be configured to render the attendees of the video conference. For example, as shown in FIG. 1C, the local attendee 120A, the local attendee 120B, the remote attendee 115B, and the remote attendee 115C are rendered on the display 155.

[0019] In some embodiments, the video conferencing system 150 can be a computing system (e.g., a laptop or desktop computer) in a home or home office. Thus, the video conferencing system 150 may not be configured to perform all of the hybrid video conferencing functions described herein. For example, the video conferencing system 150 may not include all of the hardware associated with the video conferencing system 100. For example, the video conferencing system 150 may not include multiple cameras and / or multiple displays. However, the video conferencing system 150 can be configured to perform the functions described herein, including software and / or hardware sufficient for the video conferencing system 150 to operate.

[0020] For example, the video conferencing system 150 may be configured to receive at least one video stream containing content associated with at least one attendee (e.g., each containing content). For example, the video conferencing system 150 may be configured to receive instructions on the display order of attendees in a video conference. For example, the video conferencing system 150 may be configured to render at least one received video stream based on the display order.

[0021] For example, as shown in Figures 1A and 1B, the attendees of the video conference are displayed in the order of remote attendee 115A, remote attendee 115B, remote attendee 115C, local attendee 120B, and local attendee 120A (e.g., clockwise) or remote attendee 115B, local attendee 120A, remote attendee 115B, local attendee 120B, and remote attendee 115C (e.g., left to right), with remote attendees 115B, 115B, and 115C being rendered on displays 105A, 105B, 105C, and 105D (as shown), and local attendees 120A and 120B looking at displays 105A, 105B, 105C, and 105D (e.g., sitting at the conference table) (as shown).

[0022] Figure 2 shows a top view of a photographic representation of a video conferencing system according to at least one exemplary embodiment. As shown in Figure 2, the video conferencing system 100 further includes a table 205 and lighting 210A, 210B, 210C (e.g., light-emitting diode (LED) lighting). Furthermore, display 105A shows remote attendees 115A and 115D, and display 105C shows remote attendees 115C and 115E. In some embodiments, local attendees 120A, 120B can be positioned on one side of the table 205, and the video conferencing system 100 can be positioned on the other side of the table 205. Remote attendees 115A, 115B, 115C, 115D, 115E can be displayed on displays 105A, 105B, 105C. As shown in Figure 2, two (or more) remote attendees 115A, 115B, 115C, 115D, and 115E can be displayed on displays 105A, 105B, and 105C.

[0023] In some embodiments, the display of remote attendees 115A, 115B, 115C, 115D, and 115E can be modified to simulate local presence. For example, the height of the faces of remote attendees 115A, 115B, 115C, 115D, and 115E may be displayed at the same level as local attendees 120A and 120B (e.g., appearing to be at the eye level of local attendees 120A and 120B). For example, part of the torso of remote attendees 115A, 115B, 115C, 115D, and 115E may be displayed above table 205 (e.g., appearing to be seated at table 205). In other words, the features associated with remote attendees 115A, 115B, 115C, 115D, and 115E (sometimes called presence enhancement features) can be modified to simulate local attendance during a meeting (e.g., a gathering). These features (for example, presence enhancement features) may include at least one of the following: • Natural scale rendering of attendees: Remote attendees can be shown life-size to enhance their presence. In other words, people potentially associate size with importance. Therefore, displaying remote attendees at the same size as local attendees can ensure that attention is drawn to all attendees. Compared to standard video conferencing systems, scaling up remote attendees facilitates easier reading of body language and facial expressions, thus improving social signaling. • Normalized eye level: In addition to accurately scaling remote attendees, video feeds can be dynamically positioned higher or lower to match the eye level of remote attendees with that of local attendees. Head position can be a signal of human dominance / subordination. Therefore, if local and / or remote attendees appear lower around a table, it may be unconsciously interpreted as a sense of inequality. The effect of eye level normalization can be applied to both video feeds containing a single person and video feeds within a conference room by vertically shifting each video section corresponding to each person. • Eye-line cues require careful attention to system geometry and rendering: Studies have shown that when humans detect eye-line cues, they may be sensitive to errors in horizontal angles. However, a downward eye-line angle of 5-10 degrees is acceptable. Therefore, the peripheral device strips 110A, 110B, and 110C, including the cameras, can be positioned slightly above the edges of the displays to minimize the vertical angle. Furthermore, the displays 105A, 105B, and 105C can be positioned as low as possible (relative to the table 205) to keep the distance between the head and the cameras small. In addition, the camera's eye-line angle can be minimized by ensuring sufficient depth of the table 205. In some embodiments, machine learning algorithms can be used to adjust the position of remote attendees on the displays so that they remain precisely centered directly below their respective cameras, even if remote attendees move horizontally during the meeting. • Segmentation: Image segmentation can be used to remove the background from a video feed and render remote attendees on a shared background. Rendering remote attendees on a shared background can enhance their presence because they share the background and do not have their own individual backgrounds. Rendering remote attendees on a shared background can reduce the visual clutter caused by the backgrounds of various rooms, allowing attention to be focused on the attendees themselves and the conversations they are having. This allows the meeting to scale to accommodate more attendees without becoming visually overwhelming. In addition to segmenting individual remote attendees, it is also useful to remove the backgrounds of other rooms participating in the meeting. Furthermore, the background color, pattern, and brightness can be selected from neutral to warm tones with minimal visual detail. Because of the large display area, a background that is too bright can be visually overwhelming, while one that is too dark may result in noticeable glare and diminish their presence, depending on the display technology. The brightness and color temperature of the display should be calibrated to match how attendees in the room appear, taking into account the current room lighting. • Segmentation supports continuous auto-scaling and auto-centering: If the original background is included in a continuously dynamically scaled video feed, it can result in a distracting zoom effect. Similarly, panning horizontally to keep a subject directly below each camera can result in unwanted background movement. Using segmentation allows for faster and more responsive positioning of remote attendees. This improves gaze cues and eliminates the risk of overlapping with adjacent participants as they move, even when people are densely positioned on the display. • Auto-scaling video feeds enhance presence across various remote hardware: By using face detection, the interpupillary distance of remote attendees in each video feed can be measured, and a scaling factor can be derived from this to display remote attendees in the room at actual size. Some embodiments can support any remote camera and setup. Therefore, relying on optical setups or seating distances for accurate size measurement can be an inefficient use of available resources. In some embodiments, variations in IPD among attendees can lead to scaling errors (e.g., attendees with smaller-than-average IPDs may be scaled up more inaccurately than necessary). Therefore, machine learning (ML) models can be trained to evaluate the scale (e.g., using IPD and other size cues) and correct the scaling to a more desirable level. Furthermore, face and body detection can be used to detect each remote user in the video feed from a remote meeting room, and individual scaling and repositioning can be applied (e.g., using ML models) to achieve natural scale rendering. • Edge processing for torso cropping and other video clip artifacts: Remote attendees may not be fully captured by their local camera. Normalizing eye level can create a gap between the lower edge of the video feed and the table surface. The position of this edge changes dynamically depending on factors such as the distance of the remote attendee from the local camera. Sharp, moving boundaries can be distracting (e.g., as in the display of a cropped video feed). The magnitude of the movement can depend on both the amount the remote attendee moves during the meeting and the adjustment of filtering for automatic scaling and positioning of the video feed. Some embodiments can be configured to fade out the lower edge of segmented remote participants, thereby making such fluctuations virtually invisible. In some embodiments, the fade processing can have a curved shape positioned relative to the user's face and body, thereby making the cropping appear more deliberate. The size of the fade area can be a set parameter. For example, if it is too small, the movement of the edge may be visible. If it is too large, the rendered remote attendee may appear ghostly, which can reduce their presence. Similarly, fading the left and right edges of the clipping can make it appear less noticeable and more intentional. When clipping along the top edge of a video feed, it's close to the face, and a larger fade can look distracting, so a smaller fade distance may be beneficial. • Multi-view video routing infrastructure: Various architectures can be used to meet the requirement that each remote attendee receives a specific video feed. For example, a fully interconnected peer-to-peer topology with WebRTC implemented in a web application can be used. In this design, upon joining a meeting, each client connects directly to all other clients in the meeting and uses that connection to directly transmit video streams from the appropriate cameras among attendees. Another architecture uses a cloud-hosted media router, in which case each client sends all video feeds to the router along with stream identifiers, and then each client subscribes to the appropriate video stream by requesting the stream with the correct stream ID. • Multiple video cameras and spatially consistent mapping for attendees: Each display may have multiple cameras along its upper edge (e.g., within peripheral device strips 110A, 110B, 110C). In some embodiments, each display may have three (3) cameras, and the entire system (using three (3) displays or one (1) wide display) may have a total of nine cameras. Each remote attendee can receive a video feed from a camera positioned above their rendering position. This can create accurate first-person gaze cues. For example, if a local attendee looks at the position where a remote attendee is displayed in order to speak with them, the remote attendee can see from their video feed that the local attendee is looking towards that remote attendee (e.g., directly at them). • Display mounting: The display can be mounted flush against the table so that its bottom edge is below the tabletop. This mounting configuration creates the illusion that remote attendees are sitting around the table with local attendees, rather than appearing as if they were on a television screen, as is the case with traditional wall-mounted video conferencing setups. The table can conceal the edges of the display. Therefore, local attendees may experience the illusion that the legs of remote attendees are mentally filled in, similar to how one might perceive an in-room attendee across a table. • Display Space Allocation: Display space can be allocated based on the number of remote attendees in the video feed. In hybrid meetings, it is common for one remote attendee to be in front of the camera. Using endpoint metadata or face / body detection, some embodiments can be configured to reduce the display allocated to a remote attendee (either half or one-third of the display) when a single remote attendee is detected. In some embodiments, the horizontal display area for remote attendees can be reduced while maintaining the natural scale and equal eye level of attendees. This allows the system to scale to accommodate more people than simply displaying a 16:9 wide aspect ratio image. Also, a compact layout feels more natural as it is closer to the usual arrangement of people around a real table. Furthermore, less horizontal space used reduces head rotation in small meetings, making it easier to gauge social reactions. • Localized audio rendering: Each display can have two speakers positioned in the upper left and upper right corners, and a three-display system can have a total of six speakers. Audio for each remote attendee is appropriately panned between the two speakers directly above that person, creating the illusion that the remote attendee's voice is coming from the position where they are rendered. Because vertical audio localization in human perception is adaptive, the visual stimuli of the video lock the sound into its rendering position. This enhances presence and makes remote attendees feel as if they are sitting at the table with the local attendees. Furthermore, audio localization makes it easier for local attendees to handle the overlapping of utterances that often occur in fast-paced, natural conversations. • Multiple cameras and localized audio enable larger, more immersive display setups: Some embodiments can include three large displays that are physically closer together than in typical systems. This results in much greater head rotation. When only one camera is used (e.g., one centrally positioned camera), attendees shown on the far left and far right receive unnatural gaze cues because the person speaking to them has turned their head significantly to the side, even when being spoken to directly. Also, if the audio is not localized, a multi-display setup with a wide field of view can be distracting and annoying, as it requires constant head movement to "find" the person currently speaking. • Conference Table: The table shape can be selected to suit social dynamics and camera placement. For example, some embodiments use multiple cameras. Therefore, some embodiments may have the constraint that each camera must be able to keep all attendees in its field of view. Displays can be mounted on the table. Therefore, a curved shape allows all local attendees to see all remote attendees in a good, non-oblique view. Exemplary tables may include a curve at the back edge to achieve both of these design goals. In some embodiments, the table depth can be adjusted to position remote attendees in social space beyond their personal space, which could cause social discomfort and fatigue. Furthermore, the curve in front of where local attendees sit can be configured to maintain the social dynamics of the room, allowing attendees to see each other without interfering with each other's view. This creates a balanced hybrid meeting experience in which remote and local attendees can see and be seen equally among all attendees. • Interoperability: Some embodiments can operate with existing remote video conferencing hardware. In some embodiments, remote attendees may not need to install any special equipment. Some embodiments can operate with existing webcams and computers, and can compensate for different fields of view and distances to the camera by using scaling and positioning techniques on the video feed. • Different number of displays: Some embodiments are described as using three (3) displays. However, some embodiments can be scaled to support one (1), two (2), four (4), five (5), or more displays. • Remote System: A remote system with an ultrawide curved display (see Figure 1B) and multiple cameras can be used. In some embodiments, an ultrawide monitor can be used to allow remote users to have an experience closer to the local experience. The wide width of the display allows remote attendees to be positioned within a virtual circle that they would see in the room. In this embodiment, multiple video cameras can also be positioned evenly or at different heights. This also allows remote attendees to send gaze cues to attendees in the room and other remote attendees.

[0024] In some embodiments, the video stream transmitted to remote attendees 115A, 115B, 115C, 115D, and 115E can be streamed with a spatially consistent mapping (e.g., including the display order or the order in which attendees are displayed). In other words, the mapping, display order, or rendering position of remote attendees 115A, 115B, 115C, 115D, and 115E can be consistent across local and remote video conferencing systems. In some embodiments, the video conference can include one local video conferencing system and at least one remote video conferencing system. As shown in Figure 2, in a counterclockwise display order, the display order may be remote attendee 115E, remote attendee 115C, remote attendee 115B, remote attendee 115D, remote attendee 115A, local attendee 120A, local attendee 120B, and local attendee 120C. As shown in Figure 2, the left-to-right mapping, display order, or rendering position of the remote attendees is remote attendee 115A, remote attendee 115D, remote attendee 115B, remote attendee 115C, and remote attendee 115E. Furthermore, the positions (or physical positions) of the local attendees are such that local attendee 120A is between remote attendees 115A and 115D, local attendee 120B is between remote attendees 115D and 115C, and local attendee 120C is between remote attendees 115C and 115E.

[0025] Therefore, the mapping, display order, or rendering position of attendees may be (or in that order) remote attendee 115A, local attendee 120A, remote attendee 115D, local attendee 120B, remote attendee 115C, local attendee 120C, and remote attendee 115E. This mapping, display order, or rendering position must be consistent across all video conferencing systems during the video conference. Note that on remote video conferencing systems, remote attendees may not be rendered, but the remaining attendees must be rendered in the above positions (or order).

[0026] As described above with respect to Figure 1C, the video conferencing system 150 can be configured to perform functions that are possible with the sufficient hardware it possesses. Therefore, the video conferencing system 150 can be configured to perform rendering with spatially consistent mapping (e.g., display order), rendering with natural scaling, eye line normalization, rendering normalization, rendering with a shared background, rendering with torso cutoff and edge processing, consistent display spacing (e.g., tile arrangement described later), audio localization, and the like.

[0027] Figure 3 shows a front block view of a video conferencing system display according to at least one exemplary embodiment. In some embodiments, a user interface (UI) can be rendered on the video conferencing system display. In some embodiments, the video conferencing system may include at least two displays. In some embodiments, one UI can be rendered across at least two displays. In some embodiments, at least two UIs can be rendered on associated displays. The UI can be divided into at least two display portions. In some embodiments, a first display portion and a second display portion can be rendered on a single display. In some embodiments, the first display portion can be rendered on a first display, and the second display portion can be rendered on a second display. Hereinafter, display portions may be referred to as tiles.

[0028] As shown in Figure 3, the video conferencing system 100 includes displays 105A, 105B, and 105C, each of which includes tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345. As shown in Figure 3, the video conferencing system 100 also includes peripheral device strips 110A, 110B, and 110C, each of which includes three (3) cameras 350. The cameras 350 can be positioned at the top center of each of the tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345.

[0029] Tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 can be configured to display remote attendees in a hybrid video conference. For example, remote attendee 115A may be rendered on tile 305, and remote attendee 115D may be rendered on tile 310. In some embodiments, during a hybrid video conference, tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 may not display remote attendees. For example, in some embodiments, during a hybrid video conference, tiles 305 and 310 may contain remote attendees, while tiles 315 and 320 may not display remote attendees. In other words, tiles 315 and 320 may be empty or render nothing. In some embodiments, during a hybrid video conference, tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 can be configured to display content. For example, during a hybrid video conference, tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 can display content as content displayed on a computing device or as a screencast from a computing device. For example, during a hybrid video conference, tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 can display a whiteboard showing slides(s) containing text and / or images.

[0030] In some embodiments, during a hybrid video conference, at least two of the tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 can be configured to display content. For example, during a hybrid video conference, tiles 320 and 325 can display content. In some embodiments, during a hybrid video conference, at least two of the tiles 305, 310, 315, 320, 325, 330, 335, 340, and 345 on displays 105A, 105B, and 105C can be configured to display content, while the other tiles on displays 105A, 105B, and 105C can display remote attendees.

[0031] For example, remote attendee 115A may be rendered on tile 305, and remote attendee 115D may be rendered on tile 310. In some embodiments, rendering remote attendees 115A and / or remote attendee 115D may include normalizing the presentation of remote attendees 115A and / or remote attendee 115D relative to local attendees (e.g., local attendees 120A and / or local attendee 120B). Normalizing the presentation of remote attendees may include scaling the presentation of remote attendees. For example, rendering remote attendees 115A and / or remote attendee 115D may include increasing the size of the presentation within and between tiles 305 and / or 310 to match the attendee's size and line of sight. For example, rendering remote attendees 115A and / or remote attendee 115D may include increasing the size of the presentation to approximate the size of local attendees. In other words, the rendered remote attendees 115A and / or 115D appear to be in the same room as the local attendees.

[0032] In some embodiments, rendering remote attendees 115A and / or remote attendees 115D may include normalizing the presentation of remote attendees 115A and / or remote attendees 115D, including segmenting the remote attendees and displaying them on a common background. For example, segmenting remote attendees 115A and / or remote attendees 115D may include receiving frames of streaming video and removing the background related to remote attendees 115A and / or remote attendees 115D. Segmenting an image or frame of video may include dividing the image or frame into at least two regions, partitions, segments, etc. In some implementations, a trained machine learning model may be used to segment the image or frame.

[0033] In some embodiments, segmenting remote attendees 115A and / or remote attendees 115D may include dividing a frame containing remote attendees 115A and / or remote attendees 115D into an area containing remote attendees 115A and / or remote attendees 115D and an area containing the background. Segmenting remote attendees 115A and / or remote attendees 115D may also include removing the background, leaving only remote attendees 115A and / or remote attendees 115D. In some embodiments, a common background may be added to the segmented remote attendees 115A and / or remote attendees 115D to generate a frame containing remote attendees 115A and / or remote attendees 115D on top of the common background. Since remote attendees share a background and do not individually have their own backgrounds, the common background may be a shared background configured to enhance presence. Rendering remote attendees onto a shared background reduces the visual clutter of various room backgrounds, allowing for a visual focus on the attendees themselves and the conversations taking place among both local and remote participants.

[0034] In some embodiments, rendering remote attendees 115A and / or remote attendees 115D may include normalizing the presentation of remote attendees 115A and / or remote attendees 115D, which involves shifting the presentation of remote attendees 115A and / or remote attendees 115D based on the eye position of the local attendee. For example, shifting the presentation of remote attendees 115A and / or remote attendees 115D may include moving the presentation of remote attendees 115A and / or remote attendees 115D up or down so that the local attendee is looking at the eyes of remote attendees 115A and / or remote attendees 115D when rendered. In some embodiments, shifting the presentation of remote attendees 115A and / or remote attendees 115D may include moving the presentation of remote attendees 115A and / or remote attendees 115D left or right so as to center the presentation in the associated tile. In some embodiments, shifting the presentation of remote attendees 115A and / or remote attendees 115D may include fading the boundaries of the remote attendee presentation. For example, moving a remote attendee upwards may result in torso clipping and / or other video clipping because there are no pixels to render. The fade may include expanding pixels and brightening pixels to extend the missing portion, such as the torso of the remote attendee.

[0035] In some embodiments, the video stream transmitted to remote attendees can be streamed using spatially consistent mapping. In other words, the display order or rendering position of remote attendees can be consistent across local and remote video conferencing systems. Furthermore, tiles containing content, empty tiles, etc., can be included in the mapping, display order, or rendering position. The mapping, display order, rendering position, and / or rendering order can be determined by the local video conferencing system. In some embodiments, if the video feed can be captured by a camera associated with the remote attendees, local attendees can be displayed together within the video feed. In other words, a camera 350 located above the tile presented to the remote attendees can be used to capture local attendees and transmit it as a video feed to the remote attendees. Thus, local attendees can be rendered in the same position on the remote attendees' displays. If there are more than one local attendee, the group of local attendees can be displayed together in the same position on the remote attendees' displays. In some embodiments, local attendees can be segmented (e.g., using an ML model) and communicated individually within the video stream. In this embodiment, local attendees are captured using cameras associated with remote attendees and segmented after capture. In this embodiment, segmentation may include identifying local attendees and removing the background and other local attendees from the video feed.

[0036] In some embodiments, hybrid video conferencing may include and / or add surplus remote attendees. For example, in some embodiments, the number of attendees may exceed the space on the display(s) required to render them. Therefore, the number of tiles may not be sufficient to present all attendees. Thus, some attendees may be identified as surplus attendees. Surplus attendees may be identified when the video conference is started and / or added to the video conference after it has started (e.g., attendees joining late). Furthermore, if a surplus attendee is the current speaker, the surplus attendee may be repositioned. For example, if a surplus attendee is the active speaker, the surplus attendee may be swapped with a remote attendee (their tile position is swapped). Therefore, a surplus remote attendee is a remote attendee who is not associated with a tile, and the tile cannot be added to the display. In some embodiments, a tile may be divided into at least two tiles. In some embodiments, a tile may be divided vertically. In some embodiments, a tile may be divided horizontally. For example, as shown in Figure 3, tile 340 can be divided to produce tile 340' and tile 340''.

[0037] The above UI may include additional features useful for the video conferencing system. For example, the UI may include a calendar function used to schedule video conferences. For example, the UI may include a prompt to wait for attendees to join. For example, the UI may include user animations (e.g., fade in, fade out, etc.) when attendees join or leave. For example, the UI may include a request to join function. For example, the UI may include a non-video (e.g., phone) attendee view (e.g., tiles). For example, the UI may include identification functions (e.g., attendee name, location, email, etc.). For example, the UI may include an attention function (e.g., raise hand). For example, the UI may include functions such as mute and / or camera off, and avatars. For example, the UI may include control icons for the included functions. For example, the UI may include a prompt for the current speaker. For example, the UI may include presentation and recording utilities. Other video conferencing functions and / or utilities are within the scope of this disclosure.

[0038] Figure 4 shows a block diagram of a video conferencing system according to at least one exemplary embodiment. As shown in Figure 4, the video conferencing system 100 includes displays 105A, 105B, 105C and cameras 350A, 350B, 350C. The video conferencing system 100 further includes hubs 405A, 405B, 405C, speakers 410A, 410B, 410C, a microphone 415, an audio amplifier 420, an audio processor 425, a computing device 430, a power supply 435, and a server 440. In some embodiments, the audio amplifier 420, the audio processor 425, the computing device 430, and the power supply 435 can be mounted as components of an equipment rack 445.

[0039] As described above, displays 105A, 105B, and 105C can be configured to display a user interface (UI) associated with a hybrid video conference. The UI can be configured to display tiles on displays 105A, 105B, and 105C. The tiles can be used to render remote attendees of the video conference. In some embodiments, the UI can include three (3) tiles for each of displays 105A, 105B, and 105C. For example, display 105A can display three (3) tiles, display 105B can display three (3) tiles, and display 105C can display three (3) tiles. Thus, in some embodiments, the UI can be configured to display a total of nine (9) tiles.

[0040] As described above, each tile can be used to render at least one remote attendee, to be empty, to render content, etc. Furthermore, each tile can be associated with cameras 350A, 350B, and 350C. In Figure 4, cameras 350A, 350B, and 350C each contain three (3) cameras. In some embodiments, cameras 350A, 350B, and 350C can be contained within peripheral device strips 110A, 110B, and 110C. For example, display 105A can display three (3) tiles, the left tile can be associated with the left camera of camera 350A, the center tile can be associated with the center camera of camera 350A, and the right tile can be associated with the right camera of camera 350A. When remote attendees are rendered within the tile associated with display 105A, audio associated with the remote attendees can be generated using speaker 410A. If a remote attendee is rendered within a tile associated with display 105B, the audio associated with the remote attendee can be generated using speaker 410B. If a remote attendee is rendered within a tile associated with display 105C, the audio associated with the remote attendee can be generated using speaker 410C. By generating audio for speakers 410A, 410B, and 410C associated with displays 105A, 105B, and 105C, the audio can be localized to the remote attendee rendered in the tiles of displays 105A, 105B, and 105C. In some embodiments, speakers 410A, 410B, and 410C can be included in peripheral device strips 110A, 110B, and 110C. For example, localizing audio may include stereo panning the remote attendee's audio feed based on the rendering position of the remote attendee and the positions of 410A, 410B, and 410C. Audio localization may depend on the geometry of the video conferencing system.For example, if the left speaker is directly above the left remote attendee and the right speaker is above the right attendee, the panning could be, for example, 100% to the left speaker of the left remote attendee, 100% to the right speaker of the right remote attendee, and 50% to each speaker of the center remote attendee.

[0041] Server 440 can be configured to communicate and receive video streams associated with a hybrid video conference. Server 440 can be a computing device with multiple inputs and multiple outputs. Server 440 can be configured to receive video streams from remote attendees. Furthermore, Server 440 can be configured to transmit video streams to remote attendees. Audio processor 425 can be configured to process audio received from microphone 415 in order to transmit audio to remote attendees. Audio processor 425 can be configured to process audio received from remote attendees via server 440, and audio amplifier 420 can be configured to amplify this audio for playback on speakers 410A, 410B, and 410C. The audio can be localized to remote attendees rendered on tiles on displays 105A, 105B, and 105C by playing the audio on one of the associated speakers 410A, 410B, and 410C.

[0042] In some embodiments, cameras 350A, 350B, and 350C can be used to generate video of local attendees for a video stream transmitted to remote attendees. In some embodiments, cameras 350A, 350B, and 350C can generate video and communicate that video to computing device 430 via hubs 405A, 405B, and 405C. Hubs 405A, 405B, and 405C can be a three-to-one hub to combine the video feeds of the three individual cameras 350A, 350B, and 350C. Power supply 435 can be configured to supply power to at least computing device 430.

[0043] The computing device 430 can be configured to provide processing resources for the hybrid video conferencing functionality described herein. For example, the computing device 430 can be configured to generate a video stream of local attendees for transmission to remote attendees. For example, the computing device 430 can be configured to render the video of remote attendees onto tiles displayed on displays 105A, 105B, and 105C. In some embodiments, the video stream transmitted to remote attendees may be streamed using a spatially consistent mapping. In other words, the display order or rendering position of remote attendees may be consistent across the local and remote video conferencing systems. Furthermore, tiles containing content, empty tiles, etc., may be included in the mapping, display order, or rendering position. The mapping, display order, rendering position, and / or rendering order may be determined by the computing device 430 of the local video conferencing system.

[0044] Figure 5 shows a rear block diagram of a display according to at least one exemplary embodiment. As shown in Figure 5, several components of the video conferencing system can be mounted on the display 105. For example, a peripheral device strip 110A including a camera 350 and a speaker 410 can be mounted on the display 105. A cover glass 505 can be mounted on the display 105. The cover glass 505 can be mounted on the display 105 and / or superimposed on the display 105 (e.g., superimposed on the top edge). The cover glass 505 can be configured to protect the camera 350 and the speaker 410. The cover glass 505 can be transparent or substantially transparent. The cover glass 505 can be configured to be hidden, inconspicuous, and / or not distracting to attendees. For example, lighting 210 can be mounted on the display 105. In some embodiments, lighting 210 can be LEDs.

[0045] Figures 6A, 6B, and 6C show block diagrams of a multi-camera display arrangement according to at least one exemplary embodiment. As shown in Figures 6A-6C, displays A+1, ..., XT, ..., X, ...X+T, ..., A-1 each have associated cameras 605-1, ..., 605-n. In some embodiments, the multi-camera display arrangement can maintain eye-tracking cues for representative remote attendees of $n$ attendees in a video conferencing system. In some embodiments, remote attendees can be rendered in the same order on all attendee displays (or tiles) having associated cameras 605-1, ..., 605-n. In some embodiments, local attendee A can be in a hybrid video conference with n remote attendees rendered on displays A+1, ..., XT, ..., X, ...X+T, ..., A-1. In some embodiments, displays A+1, ..., XT, ..., X, ...X+T, ..., A-1 can be implemented as tiles as described above.

[0046] When local attendee A views the remote attendees rendered on display X, the remaining remote attendees in the hybrid video conference can see local attendee A, who is facing the direction of the remote attendees rendered on display X, on their respective remote displays.

[0047] For example, a remote attendee rendered on the display to the left of display X (e.g., displays A+1, ..., XT, ..., identified as section 1 in Figure 6B) can see local attendee A looking to the left from the camera 605-1, ..., 605-n associated with each remote attendee. Remote attendees can be rendered at all positions in the video conference based on the same order as displays A+1, ..., XT, ..., X, ..., X+T, ..., A-1. Therefore, when remote attendee X is rendered on a remote attendee's display, it is rendered in front of local attendee A, so on the local display of any remote attendee (e.g., a remote attendee rendered on display XT), remote attendee X will appear to the left of local attendee A. Thus, each remote attendee can see local attendee A looking in the direction of remote attendee X.

[0048] For example, a remote attendee rendered on the display to the right of display X (e.g., displays ..., X+1, ..., A-1, identified as Section 2 in Figure 6C) can see local attendee A looking to the right from the camera 605-1, ..., 605-n associated with each remote attendee. Remote attendees can be rendered at all positions in the video conference based on the same order as displays A+1, ..., XT, ..., X, ..., X+T, ..., A-1. Therefore, when remote attendee X is rendered on the remote attendee's display, it is rendered after local attendee A, so on the local display of any remote attendee (e.g., a remote attendee rendered on display X+1), remote attendee X will appear to the right of local attendee A. Thus, each remote attendee can see local attendee A looking in the direction of remote attendee X.

[0049] As described above, local attendees can be associated with individual video feeds, and the mapping or display order can be linear (e.g., left to right or right to left). In this embodiment, local attendees (e.g., local attendee A) can move. In some embodiments, the mapping, display order, or rendering position can be modified. For example, if local attendee A moves to the left of remote attendee XT, the mapping, display order, or rendering position can be modified based on that movement. As shown in Figures 6B and 6C, sections 2 and 3 can be modified based on the movement of local attendee A.

[0050] Figures 6D and 6E show block diagrams of display order according to at least one exemplary embodiment. Attendees mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc., can be from one of many viewpoints. For example, attendees mapping, spatial mapping, display position, display order, display location, rendering position, rendering order, rendering location, etc., can be arranged in a clockwise or counterclockwise circular arrangement when viewed from above. Figures 6D and 6E show attendees displayed in a counterclockwise circular arrangement when viewed from above.

[0051] Referring to Figure 6D, circle 1 can represent local attendees or attendee systems, and circles 2, 3, 4, and 5 can represent remote attendees or attendee systems. Display order 610-1 shows the display order from the perspective of local attendees (circle 1), where display order 610-1 is circles 5, 4, 3, and 2 or remote attendees 5, 4, 3, and 2. In order to maintain consistency across all devices associated with the video conference, some of the above features (e.g., eye line of sight) must be the same for all attendees. Therefore, referring to Figure 6E, display order 610-2 shows the display order from the perspective of remote attendees (circle 2), where display order 610-2 is circles 3, 4, 5, and 1 or remote attendees 3, 4, 5 and local attendee 1. Display order 610-3 shows the display order from the perspective of the remote attendee (circle 3), and display order 610-3 is circles 4, 5, 1, and 2 or remote attendees 4, 5, local attendee 1, and remote attendee 2. Display order 610-4 shows the display order from the perspective of the remote attendee (circle 4), and display order 610-4 is circles 5, 1, 2, and 3 or remote attendee 5, local attendee 1, and remote attendees 2 and 3. Display order 610-5 shows the display order from the perspective of the remote attendee (circle 5), and display order 610-5 is circles 1, 2, 3, and 4 or local attendee 1, and remote attendees 2, 3, and 4.

[0052] Example 1. Figure 7 is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. As shown in Figure 7, in step S705, a first remote device associated with a first remote attendee receives a remote video stream from a second remote device containing content associated with a second remote attendee of the video conference. In step S710, the first remote device receives a local video stream from a local device associated with a local attendee of the video conference containing instructions for the display order of the first remote attendee, the second remote attendee, and the local attendee. In step S715, the first remote device renders the remote video stream and the local video stream based on the display order.

[0053] Example 2. The method according to Example 1, wherein rendering of remote and local video streams may include normalizing the presentation of a second remote and local attendee to a first remote attendee.

[0054] Example 3. The method according to Example 2, wherein the normalization of the presentations of the second remote and local attendees may include scaling the presentations of the second remote and local attendees.

[0055] Example 4. The method according to Example 2, wherein the normalization of the presentation of the second remote attendee and local attendee includes segmenting the presentation of the second remote attendee and local attendee and rendering (or displaying) the second remote attendee and local attendee against a common background.

[0056] Example 5. The method according to Example 2, wherein the normalization of the presentation of the second remote attendee and local attendee may include shifting the presentation of the second remote attendee and local attendee based on the eye position of the first remote attendee.

[0057] Example 6. The method according to Example 5, wherein the shift in the presentation of the second remote attendee and local attendee may include fading the boundary between the presentation of the second remote attendee and local attendee.

[0058] Example 7. The method according to Example 1, wherein rendering of remote video streams and local video streams may include generating a user interface having a first display portion including a second remote attendee and a second display portion including a local attendee.

[0059] Example 8. The method according to Example 7, wherein the user interface may include a third display portion having content associated with the video conference.

[0060] Example 9. The method according to Example 7, wherein the user interface may include a third display portion that is empty.

[0061] Example 10. The method according to Example 1, wherein rendering of the first and second video streams may include generating a user interface having a number of display portions based on the number of video streams associated with the video conference.

[0062] Example 11. Figure 8 is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. As shown in Figure 8, step S805 generates a user interface including a first display portion corresponding to a first camera and a second display portion corresponding to a second camera. Step S810 renders the UI on the display of the video conferencing system. Step S815 receives a video stream corresponding to a remote attendee of the video conference. Step S820 renders the remote attendee in one of the first or second display portions, the rendering including normalizing the presentation of the remote attendee to the local attendee. In some embodiments, video captured by the first camera can be streamed to the first remote attendee. Furthermore, video associated with and / or received from the first remote attendee can be rendered in the first display portion. In some embodiments, video captured by the second camera can be streamed to the second remote attendee. Furthermore, video associated with and / or received from a second remote attendee can be rendered in the second display section.

[0063] Example 12. The method according to Example 11, wherein the normalization of the remote attendee's presentation may include scaling the remote attendee's display.

[0064] Example 13. The method according to Example 11, wherein the normalization of the presentation of remote attendees may include segmenting the remote attendees' displays and displaying the remote attendees on a common background.

[0065] Example 14. The method according to Example 11, wherein the normalization of the presentation of remote attendees may include shifting the remote attendees' displays based on the eye position of the local attendees.

[0066] Example 15. The method according to Example 14, wherein shifting the remote attendee's display may include fading the border of the remote attendee's display.

[0067] Example 16. The method according to Example 11, wherein the generation of a first display portion may include centering the first display portion based on the position of a first camera, and the generation of a second display portion may include centering the second display portion based on the position of a second camera.

[0068] Example 17. The method according to Example 11, wherein the generation of the first and second display portions on the display includes generating a third display portion on the display. Alternatively (or further), the user interface may include a third display portion on the display.

[0069] Example 18. The method according to Example 17, wherein the first display portion may include remote attendees, and the second and third display portions may be empty.

[0070] Example 19. The method according to Example 17, wherein the first display portion may include remote attendees, and the second and third display portions may include content associated with the video conference (e.g., other than remote attendees and / or local attendees).

[0071] Example 20. The method according to Example 11, wherein the first camera can be configured to capture video of multiple local attendees of a video conference, and the second camera can be configured to capture video of multiple local attendees of a video conference.

[0072] Example 21. The method according to Example 11, wherein the generation of a first and second display portion on a display may include generating a plurality of display portions based on the number of video streams associated with the video conference. Alternatively (or further), the user interface may include a number of display portions based on the number of video streams associated with the video conference.

[0073] Example 22. The method according to Example 11, wherein the generation of the first and second display portions on the display may include the generation of a display portion that includes surplus remote attendees.

[0074] Example 23. The method according to Example 11, which may further include localizing the audio of remote attendees to speakers.

[0075] Example 24. The method according to Example 11, wherein the video stream may include at least two remote attendees, and the generation of the first and second display portions on the display may include generating a plurality of display portions based on at least two remote attendees. Alternatively (or further), the user interface may include a number of display portions based on at least two remote attendees.

[0076] Example 25. The method according to Example 11, further comprising determining that the video stream contains extra remote attendees, and in response to the determination that the video stream contains extra remote attendees, generating a split display portion by splitting one of the first display portion and the second display portion, and displaying the extra remote attendees in the split display portion.

[0077] Example 26. The method according to Example 25, wherein one of the first display portion and the second display portion can be divided vertically.

[0078] Example 27. The method according to Example 25, wherein one of the first display portion and the second display portion can be divided horizontally.

[0079] Example 28. The method according to Example 11, wherein the display may be a first display having three display portions, a second display having three display portions, and a third display having three display portions (or including them).

[0080] Example 29. The method according to Example 11, wherein the display may be a wide aspect ratio display including a first display portion and a second display portion.

[0081] Example 30. Figure 9 is a block diagram of a method for operating a video conferencing system according to an exemplary embodiment. As shown in Figure 9, step S905 generates a user interface including a first display portion corresponding to a first camera and a second display portion corresponding to a second camera. Step S910 renders the UI on the display of the video conferencing system. Step S915 receives a first video stream corresponding to remote attendees of the video conference. Step S890 generates a second video stream including local attendees, which includes instructions on the display order of local attendees to remote attendees.

[0082] Example 31. The method according to Example 30, wherein the video stream includes multiple local attendees, and the display order includes all local attendees together in one position.

[0083] Example 32. The method according to Example 30, wherein rendering may include normalizing the presentation of remote attendees to local attendees.

[0084] Example 33. The method according to Example 30, wherein the normalization of the remote attendee's presentation may include scaling the remote attendee's display.

[0085] Example 34. The method according to Example 30, wherein the normalization of the presentation of remote attendees includes segmenting the remote attendees' displays and displaying the remote attendees on a common background.

[0086] Example 35. The method of Example 30 may further include shifting the presentation of the remote attendee by moving the presentation of the remote attendee up or down on the display.

[0087] Example 36. The method according to Example 35, wherein the shift of the remote attendee's presentation may include fading the border of the remote attendee's display.

[0088] Example 37. The method according to Example 30, wherein the generation of a first display portion may include centering the first display portion based on the position of a first camera, and the generation of a second display portion may include centering the second display portion based on the position of a second camera.

[0089] Example 38. The method according to Example 30, wherein the generation of the first and second display portions on the display includes generating a third display portion on the display. Alternatively (or further), the user interface may include a third display portion on the display.

[0090] Example 39. The method according to Example 38, wherein the first display portion may include remote attendees, and the second and third display portions may be empty.

[0091] Example 40. The method according to Example 38, wherein the first display portion may include remote attendees, and the second and third display portions may include content associated with the video conference (e.g., other than remote attendees and / or local attendees).

[0092] Example 41. The method according to Example 30, wherein the generation of a first and second display portion on a display may include generating a plurality of display portions based on the number of video streams associated with the video conference. Alternatively (or further), the user interface may include a number of display portions based on the number of video streams associated with the video conference.

[0093] Example 42. The method according to Example 30, wherein the generation of the first and second display portions on the display may include the generation of a display portion that includes surplus remote attendees.

[0094] Example 43. The method of Example 30 may further include tracking the gaze of local attendees.

[0095] Example 44. The method of Example 30 may further include tracking the gaze of remote attendees.

[0096] Example 45. The method of Example 30 may further include localizing the audio of remote attendees to speakers.

[0097] Example 46. The method according to Example 30, wherein the video stream may include at least two remote attendees, and the generation of the first and second display portions on the display may include generating a plurality of display portions based on at least two remote attendees. Alternatively (or further), the user interface may include a number of display portions based on at least two remote attendees.

[0098] Example 47. The method of Example 30 may further include determining that the video stream contains extra remote attendees, and in response to the determination that the video stream contains extra remote attendees, generating a split display portion by splitting one of the first and second display portions, and displaying the extra remote attendees in the split display portion.

[0099] Example 48. The method according to Example 47, wherein one of the first display portion and the second display portion can be divided vertically.

[0100] Example 49. The method according to Example 47, wherein one of the first display portion and the second display portion can be divided horizontally.

[0101] Example 50. The method according to Example 30, wherein the display comprises a first display having three display portions, a second display having three display portions, and a third display having three display portions.

[0102] Example 51. The method according to Example 30, wherein the display may be a wide aspect ratio display including a first display portion and a second display portion.

[0103] Example 52. The method may include one or any combination of Examples 1 to 54.

[0104] Example 53. A non-temporary computer-readable storage medium containing instructions, wherein the instructions are stored in the non-temporary computer-readable storage medium and, when executed by at least one processor, are configured to cause a computing system to perform the method described in any one of Examples 1 to 52.

[0105] Example 54. Apparatus comprising means for performing the method described in any one of Examples 1 to 52.

[0106] Example 55. An apparatus comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to use the at least one processor to cause the apparatus to perform at least one of the methods described in any one of Examples 1 to 52.

[0107] An exemplary embodiment may include a non-temporary computer-readable storage medium containing instructions, which are stored on the non-temporary computer-readable storage medium and, when executed by at least one processor, are configured to cause a computing system to perform one of the methods described above. An exemplary embodiment may include a device that includes means for performing one of the methods described above. An exemplary embodiment may include a device having at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured, using at least one processor, to cause the device to perform at least one of the methods described above.

[0108] Various embodiments of the systems and technologies described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system, comprising at least one programmable processor, the at least one programmable processor may be dedicated or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0109] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0110] To provide user interaction, the systems and technologies described herein may be implemented on a computer having a display device (LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., mouse or trackball) on which the user can input information into the computer. Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form, including acoustic, spoken language, or haptic input.

[0111] The systems and technologies described herein may be implemented in a computing system that includes backend components (e.g., as a data server), middleware components (e.g., an application server), or frontend components (e.g., a client computer having a graphical user interface or web browser that allows a user to interact with embodiments of the systems and technologies described herein), or in a combination of such backend, middleware, or frontend components. The components of the system may be interconnected by digital data communications (e.g., communication networks) of any form or medium. Embodiments of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0112] A computing system can include clients and servers. Clients and servers are generally geographically distant from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other.

[0113] Several embodiments have been described. Nevertheless, it should be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0114] Additionally, the logic flow shown in the diagram does not require a specific sequence, i.e., a sequential order, to achieve the desired result. Furthermore, other steps may be added to the described flow, or steps may be removed from the described flow, and other components may be added to the described system, or other components may be removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

[0115] While specific features of the embodiments described herein have been illustrated, numerous modifications, substitutions, alterations, and equivalents will be conceivable to those skilled in the art. It should be understood that the appended claims are intended to encompass all such modifications and alterations that fall within the scope of the embodiments. They are presented only as examples and not as limitations, and various modifications in form and detail are permitted. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the different embodiments described.

[0116] The exemplary embodiments may include various modifications and alternative forms, which are shown as examples in the drawings and described in detail herein. However, it should be understood that the exemplary embodiments are not intended to limit themselves to any particular form disclosed, but rather should include all modifications, equivalents, and alternative forms that fall within the scope of the claims. Similar figures refer to similar components throughout the description of the drawings.

[0117] Some of the exemplary embodiments described above are explained as processes or methods shown as flowcharts. While flowcharts describe operations as sequential, many operations may occur in parallel, concurrently, or simultaneously. The order of operations may also be changed. A process may terminate when its operations are completed, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0118] Some of these are illustrated by flowcharts. The methods described above can be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment for performing the required task may be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) may perform the required task.

[0119] The specific structural and functional details disclosed herein are provided solely for illustrative purposes to illustrate exemplary embodiments. However, exemplary embodiments can be embodied in many alternative forms and should not be construed as being limited only to the embodiments described herein.

[0120] Furthermore, while terms such as “first,” “second,” etc., may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. Where used herein, the term “and / or” includes one or any combination of the relevant described items.

[0121] When an element is described as being connected to or bound to another element, it should be understood that the element can be directly connected to or bound to the other element, or that an intermediary element may exist. In contrast, when an element is described as being directly connected to or bound to another element, no intermediary element exists. Other words used to indicate relationships between elements should be interpreted similarly (e.g., between to directly, adjacent to directly adjacent, etc.).

[0122] The terminology used herein is for the purpose of describing specific embodiments and is not intended to limit exemplary embodiments. Where used herein, the singular forms a, an, and the are also intended to include the plural forms unless the context otherwise explicitly indicates. It should be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0123] Furthermore, it should be noted that in some alternative embodiments, the functions / actions shown may occur in a different order than that shown in the diagrams. For example, two diagrams shown consecutively may actually be performed in reverse order, depending on the functions / actions that can be performed simultaneously or are involved.

[0124] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the exemplary embodiments belong. For example, terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as their meanings in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0125] The exemplary embodiments described above and the corresponding embodiments for carrying out the inventions are presented with respect to software, or with respect to algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are intended to effectively convey the content of the work to those skilled in the art. An algorithm is considered, if the term is used herein and in general usage, to be a set of self-consistent steps leading to a desired result. These steps require the physical manipulation of physical quantities. Usually, but not always, these quantities take the form of optical, electrical, or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. For reasons of general use, it is sometimes convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.

[0126] In the exemplary embodiments described above, references to actions and symbolic representations of behavior (e.g., in the form of a flowchart) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types, and which may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate array (FPGA) computers, etc.

[0127] However, it should be recognized that all these terms and similar terms should correspond to appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specified or as is evident from the description, terms such as decisions to process or compute or calculate or display refer to actions and processes of a computer system or similar electronic computing device, which manipulates data represented as physical quantities, electronic quantities, in the computer system's registers and memory and converts them into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage devices, transmission devices, or display devices.

[0128] It should also be noted that the software implementations of exemplary embodiments are typically encoded on some form of non-temporary program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read-only memory, i.e., CD-ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cables, optical fibers, or any other suitable transmission medium known in the art. The exemplary embodiments are not limited to these aspects of any given embodiment.

[0129] Finally, while the attached claims describe specific combinations of features described herein, it should be noted that the scope of this disclosure is not limited to the specific combinations claimed below, but rather extends to encompass any combination of features of the embodiments disclosed herein, regardless of whether the specific combination is explicitly enumerated in the attached claims at this point.

Claims

1. A first remote device associated with a first remote attendee receives a remote video stream from a second remote device containing content associated with a second remote attendee of a video conference. The first remote device receives a local video stream from a local device associated with a local attendee of the video conference, which includes instructions for the display order of the first remote attendee, the second remote attendee, and the local attendee. On the first remote device, the remote video stream and the local video stream are rendered based on the display order, Methods that include...

2. The method according to claim 1, wherein the rendering of the remote video stream and the local video stream includes normalizing the presentation of the second remote attendee and the local attendee to the first remote attendee.

3. The method according to claim 2, wherein the normalization of the presentations of the second remote attendee and the local attendee comprises scaling the presentations of the second remote attendee and the local attendee.

4. The method according to any one of claims 2 and 3, wherein the normalization of the presentations of the second remote attendee and the local attendee comprises segmenting the presentations of the second remote attendee and the local attendee and rendering the second remote attendee and the local attendee against a common background.

5. The method according to any one of claims 2 to 4, wherein the normalization of the presentations of the second remote attendee and the local attendee includes shifting the presentations of the second remote attendee and the local attendee based on the eye position of the first remote attendee.

6. The method according to claim 5, wherein the shift of the presentations of the second remote attendee and the local attendee includes fading the boundaries of the presentations of the second remote attendee and the local attendee.

7. The method according to any one of claims 1 to 6, wherein the rendering of the remote video stream and the local video stream comprises generating a user interface having a first display portion including the second remote attendee and a second display portion including the local attendee.

8. The method according to claim 7, wherein the user interface includes a third display portion having content associated with the video conference.

9. The method according to claim 7, wherein the user interface includes a third display portion that is empty.

10. The method according to any one of claims 1 to 6, wherein the rendering of the remote video stream and the local video stream comprises generating a user interface having a number of display portions based on the number of video streams associated with the video conference.

11. The display and A first camera configured to capture video of local attendees in a video conference, A second camera configured to capture video of the local attendees of the aforementioned video conference, Processor and A video conferencing system including the processor, It is configured to generate a user interface (UI), and the UI is A first display section corresponding to the first camera, A second display section corresponding to the second camera, The processor further includes, The video stream corresponding to the remote attendees of the aforementioned video conference is received. The remote attendee is configured to be rendered in one of the first or second display portions, and the rendering includes normalizing the presentation of the remote attendee to the local attendee. Video conferencing system.

12. The video conferencing system according to claim 11, wherein the normalization of the presentation of the remote attendee includes scaling the display of the remote attendee.

13. The video conferencing system according to any one of claims 11 and 12, wherein the normalization of the presentation of the remote attendees includes segmenting the presentation of the remote attendees and displaying the remote attendees on a common background.

14. The video conferencing system according to any one of claims 11 to 13, wherein the normalization of the presentation of the remote attendee includes shifting the presentation of the remote attendee based on the eye position of the local attendee.

15. The video conferencing system according to claim 14, wherein the shift of the remote participant's display includes fading the boundaries of the remote participant's display.

16. The generation of the first display portion includes centering the first display portion based on the position of the first camera, The video conferencing system according to any one of claims 11 to 15, wherein the generation of the second display portion includes centering the second display portion based on the position of the second camera.

17. The video conferencing system according to any one of claims 11 to 16, wherein the generation of the first display portion and the second display portion on the display includes generating a third display portion on the display.

18. The first display portion includes the remote attendees, The video conferencing system according to claim 17, wherein the second display portion and the third display portion are empty.

19. The first display portion includes the remote attendees, The video conferencing system according to claim 17, wherein the second display portion and the third display portion include content.

20. The first camera captures video of multiple local attendees of the video conference, The video conferencing system according to any one of claims 11 to 19, wherein the second camera is configured to capture video of the plurality of local attendees of the video conference.

21. The video conferencing system according to any one of claims 11 to 20, wherein the generation of the first display portion and the second display portion on the display comprises generating a plurality of display portions based on the number of video streams associated with the video conferencing.

22. The video conferencing system according to any one of claims 11 to 21, wherein the generation of the first display portion and the second display portion on the display includes generating a display portion that includes surplus remote attendees.

23. The video conferencing system according to any one of claims 11 to 22, wherein the processor is further configured to localize the audio of the remote attendees to the speakers.

24. The aforementioned video stream includes at least two remote attendees, The video conferencing system according to any one of claims 11 to 23, wherein the generation of the first display portion and the second display portion on the display comprises generating a plurality of display portions based on the at least two remote attendees.

25. The aforementioned processor further, It is determined that the aforementioned video stream includes surplus remote attendees, In response to the determination that the aforementioned video stream includes surplus remote attendees, A divided display portion is generated by dividing one of the first display portion and the second display portion. The video conferencing system according to any one of claims 11 to 24, configured to display the surplus remote attendees within the split display portion.

26. The video conferencing system according to claim 25, wherein one of the first display portion and the second display portion is divided vertically.

27. The video conferencing system according to claim 25, wherein one of the first display portion and the second display portion is divided horizontally.

28. The video conferencing system according to any one of claims 11 to 27, wherein the display comprises a first display including three display portions, a second display including three display portions, and a third display including three display portions.

29. The video conferencing system according to any one of claims 11 to 28, wherein the display is a wide aspect ratio display including the first display portion and the second display portion.

30. The display and A first camera configured to capture video of local attendees in a video conference, A second camera configured to capture video of the local attendees of the aforementioned video conference, Processor and A video conferencing system including the processor, It is configured to generate a user interface (UI), and the UI is A first display section corresponding to the first camera, A second display section corresponding to the second camera, The processor further includes, Receiving a first video stream corresponding to a remote participant in the aforementioned video conference, The remote attendees are rendered in one of the first or second display sections. A video conferencing system configured to generate a second video stream including the local attendees, wherein the second video stream includes instructions on the display order of the local attendees to the remote attendees.

31. The aforementioned second video stream includes multiple local attendees, The video conferencing system according to claim 30, wherein the display order includes the multiple local attendees together in one position.

32. The video conferencing system according to any one of claims 30 and 31, wherein the rendering includes normalizing the presentation of the remote attendees to the local attendees.

33. The video conferencing system according to claim 32, wherein the normalization of the presentation of the remote attendee includes scaling the display of the remote attendee.

34. The video conferencing system according to claim 32, wherein the normalization of the presentation of the remote attendees includes segmenting the displays of the remote attendees and displaying the remote attendees on a common background.

35. The video conferencing system according to claim 32, further comprising shifting the presentation of the remote participant by moving the presentation of the remote participant up or down on the display.

36. The video conferencing system according to claim 35, wherein the shift of the remote participant's presentation includes fading the boundaries of the remote participant's display.

37. The generation of the first display portion includes centering the first display portion based on the position of the first camera, The video conferencing system according to any one of claims 30 to 36, wherein the generation of the second display portion includes centering the second display portion based on the position of the second camera.

38. The video conferencing system according to any one of claims 30 to 37, wherein the generation of the first display portion and the second display portion on the display comprises generating a third display portion on the display.

39. The first display portion includes the remote attendees, The video conferencing system according to claim 38, wherein the second display portion and the third display portion are empty.

40. The first display portion includes the remote attendees, The video conferencing system according to claim 38, wherein the second display portion and the third display portion include content.

41. The video conferencing system according to any one of claims 30 to 40, wherein the generation of the first display portion and the second display portion on the display comprises generating a plurality of display portions based on the number of video streams associated with the video conferencing.

42. The video conferencing system according to any one of claims 30 to 41, wherein the generation of the first display portion and the second display portion on the display includes generating a display portion that includes surplus remote attendees.

43. The video conferencing system according to any one of claims 30 to 42, wherein the processor is further configured to localize the audio of the remote attendees to the speakers.

44. The aforementioned first video stream includes at least two remote attendees, The video conferencing system according to any one of claims 30 to 43, wherein the generation of the first display portion and the second display portion on the display comprises generating a plurality of display portions based on the at least two remote attendees.

45. The aforementioned processor further, It is determined that the aforementioned video stream includes surplus remote attendees, In response to the determination that the aforementioned video stream includes surplus remote attendees, A divided display portion is generated by dividing one of the first display portion and the second display portion. The video conferencing system according to any one of claims 30 to 44, configured to display the surplus remote attendees within the split display portion.

46. The video conferencing system according to claim 45, wherein one of the first display portion and the second display portion is divided vertically.

47. The video conferencing system according to claim 45, wherein one of the first display portion and the second display portion is divided horizontally.

48. The video conferencing system according to any one of claims 30 to 47, wherein the display comprises a first display having three display portions, a second display having three display portions, and a third display having three display portions.

49. The video conferencing system according to any one of claims 30 to 48, wherein the display is a wide aspect ratio display including the first display portion and the second display portion.

Images