Eye contact optimization

Abstract

A system and method for conducting a video conference includes receiving a multimedia stream of a plurality of participants in a multimedia conference, the multimedia stream including an audio component and a video component and displaying video tiles of the participants on a display screen. The audio component and / or the video component of the multimedia stream are analyzed to detect a characteristic indicative of a first participant and a second participant making a first session with each other. A camera location on the computing device of the participant is identified. In response to identifying that the first participant and the second participant are performing the first session with each other, a video tile for the first participant and a video tile for the second participant are moved toward the camera position to an edge of a respective display screen.

Description

Background Art

[0001] In recent years, the use of video conferencing has become increasingly popular to enable multiple parties located at multiple remote locations to communicate and collaborate with each other. For example, the widespread availability of broadband Internet connections and inexpensive video capture and display equipment has made video conferencing technology readily accessible. Video conferencing systems can include dedicated systems with all required components packaged into a dedicated device and personal computer (PC) based systems including general purpose computers with webcams, communication software, and broadband Internet connections.

[0002] One goal of video conferencing technology is to serve as a viable alternative for face-to-face meetings. However, an obstacle to the widespread acceptance of video conferencing is that video conferencing systems are unable to promote user engagement during live video conferencing. This is partly due to the perceived lack of eye contact between participants in the video conference. In most embodiments, video cameras are placed at the periphery of the participant's display screen, and the video images of other participants are positioned at a distance from the camera on the display screen. Natural impulses are used for users to view the video images of other participants on the display screen during video conferencing, rather than viewing the camera. Therefore, even if the user may be maintaining eye contact with the video images of other participants, in the eyes of other participants, the user does not look at them at all, which creates a wrong impression that the user is either avoiding eye contact or being distracted. This also causes the user's video image to show the side or top of the user, rather than a direct view of the user's face, which can blur the user's facial expressions and gestures, thereby further limiting the ability of the user to communicate efficiently during video conferencing.

[0003] The above-mentioned problems also highlight another problem associated with video conferencing technology, namely, the inability to facilitate a back-and-forth conversation between two participants in a video conference, especially when there are a large number of participants. Video conferencing systems typically display video images of participants in tiles arranged in a predetermined pattern on a display screen based at least in part on a scheme set by the system. Video conferencing systems are typically able to identify the current speaker and change the display of the tiles associated with the current speaker in a manner that facilitates viewing by other participants, for example, by highlighting the tiles, increasing the size and / or position of the tiles, etc. As different participants speak during a video conference, the system sequentially changes the display of their tiles to ensure that the current speaker is always indicated. However, when two or more speakers are engaged in a back-and-forth conversation, the participant currently speaking can change rapidly. This in turn can result in rapid changes to the display of the current speaker on the participant's screen, which can be distracting and difficult to follow.

[0004] What is needed is a system and method for improving the ability of a video conferencing system to simulate a face-to-face meeting by facilitating eye contact and back-and-forth conversation between participants in the video conference. Summary of the invention

[0005] In one general aspect, the present disclosure presents a method for conducting a video conference with multiple participants, wherein each of the participants utilizes a device for sending a multimedia feed of the participant associated with the device and receiving multimedia feeds from devices associated with other participants in the video conference. The method includes: receiving the multimedia feed from multiple devices associated with each of the participants in the video conference, the multimedia feed including an audio component and a video component; displaying tiles on a display screen of a device of each of the participants, each of the tiles respectively showing the video component of one of the multimedia feeds in the multimedia feed; analyzing at least one of the audio component and the video component of multiple multimedia feeds to detect a characteristic indicating that a first participant and a second participant are conducting a first conversation with each other; identifying a first camera position of a camera on the device of the first participant and a second camera position of a camera on the device of the second participant; in response to identifying that the first participant and the second participant are conducting the first conversation with each other: moving the tile for the first participant on the display screen of the device of the second participant from a first position to a second position, the second position being closer to the second camera position than the first position; and moving the tile for the second participant on the display screen of the device of the first participant from a third position to a fourth position, the fourth position being closer to the first camera position than the third position.

[0006] In another general aspect, the present disclosure presents a data processing system having a processor and a memory in communication with the processor, wherein the memory stores executable instructions that, when executed by the processor, cause the data processing system to perform multiple functions. The functions may include: receiving multimedia streams of multiple participants in a multimedia conference, the multimedia streams including an audio component and a video component; displaying video tiles on a display screen of a device of at least one of the participants, each of the video tiles showing the video component of one of the multimedia streams; analyzing the audio component and / or the video component of the multimedia stream to detect characteristics indicating that a first participant and a second participant are conducting a first conversation with each other; identifying a camera position of a camera of the device; and in response to identifying that the first participant and the second participant are conducting the first conversation with each other, moving the video tile for the first participant and the video tile for the second participant to an edge of the display screen in the direction of the camera position.

[0007] In another general aspect, the present application describes a non-transitory computer-readable medium having instructions stored thereon, which, when executed, cause a programmable device to perform the following functions: receive multimedia streams of multiple participants in a multimedia conference, the multimedia streams including an audio component and a video component; display video tiles on a display screen of a device of at least one of the participants, each of the video tiles showing the video component of one media stream in the multimedia stream; analyze the audio component and / or the video component of the multimedia stream to detect characteristics indicating that a first participant and a second participant are conducting a first conversation with each other; identify a camera position of a camera of the device; and in response to identifying that the first participant and the second participant are conducting the first conversation with each other, move the video tile for the first participant and the video tile for the second participant to an edge of the display screen in the direction of the camera position.

[0008] This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are only used as examples and not as limitations, and depict one or more implementations according to the present teachings. In the accompanying drawings, like reference numerals refer to the same or similar elements. In addition, it should be understood that the accompanying drawings are not necessarily drawn to scale.

[0010] Figure 1 An example system is depicted upon which aspects of the disclosure may be implemented.

[0011] Figure 2 An example client device including a multimedia conferencing application with eye contact optimization according to the present disclosure is depicted.

[0012] Figure 3A-3B An example arrangement of video tiles of participants in a multimedia conference is shown.

[0013] Figure 4 Shown for use with Figure 2 An exemplary eye contact optimization system for use with a multimedia conferencing system.

[0014] Figure 5A-Figure 5C Shows that when Figure 4 Example placement of video tiles based on camera position when the eye contact optimization system detects a conversation.

[0015] Fig. 6A Shows Figure 4 An example implementation of a conversation detection module of an eye contact optimization system.

[0016] Figure 6B Shows Figure 4 An example implementation of a camera position module of an eye contact optimization system.

[0017] Figure 6C Diagram showing the relationship between head orientation / gaze direction and camera position / direction.

[0018] Figure 7 is to use Figure 4 A flow chart of a method of an eye contact optimization system for conducting video conferencing.

[0019] Figure 8 is a block diagram illustrating an example software architecture, various portions of which may be used in conjunction with the various hardware architectures described herein.

[0020] Fig. 9 is a block diagram illustrating components of an example machine configured to read instructions from a machine-readable medium and perform any of the features described herein. DETAILED DESCRIPTION

[0021] In the detailed description below, numerous specific details are set forth in order to provide a thorough understanding of the relevant teachings. It will be apparent to those of ordinary skill in the art after reading this specification that various aspects can be practiced without such details. In other cases, well-known methods, processes, components and / or circuits have been described at a relatively high level but without detail in order to avoid unnecessarily obscuring various aspects of the present teachings.

[0022] One goal of video conferencing technology is to serve as a viable alternative for face-to-face meetings. However, an obstacle to the widespread acceptance of video conferencing is that video conferencing systems are unable to promote user engagement during live video conferences. This is partly due to the perceived lack of eye contact between participants in a video conference. In most embodiments, a video camera is placed at the periphery of a participant's display screen, and the video images of other participants are positioned on the display screen at a distance from the camera. Natural impulses are used for users to view the video images of other participants on the display screen during a video conference, rather than viewing the camera. Therefore, even if a user may be maintaining eye contact with the video images of other participants, it appears to other participants that the user is not looking at them at all, which creates a false impression that the user is either avoiding eye contact or being distracted. This also results in the user's video image showing the side or top of the user, rather than a direct view of the user's face, which can blur the user's facial expressions and gestures, thereby further limiting the user's ability to communicate efficiently during a video conference.

[0023] Another problem associated with video conferencing technology is the inability to facilitate a back-and-forth conversation between two participants of a video conference, especially when there are a large number of participants. Video conferencing systems typically display video images of participants in tiles arranged in a predetermined pattern on a display screen based at least in part on a scheme set by the system. Video conferencing systems are typically able to identify the current speaker and change the display of the tiles associated with the current speaker in a manner that facilitates viewing by other participants, for example, by highlighting the tiles, increasing the size and / or position of the tiles, etc. As different participants speak during a video conference, the system sequentially changes the display of their tiles to ensure that the current speaker is always indicated. However, when two or more speakers are engaging in a back-and-forth conversation, the participant currently speaking can change rapidly. This in turn can result in rapid changes to the display of the current speaker on the participants' screens, which can be distracting and difficult to follow.

[0024] Lack of eye contact and lack of ability to effectively see and understand non-verbal cues during a video conference can result in a lack of participation in the video conference. This lack of participation can result in inefficiency because it may take more time and effort to effectively convey information between the parties. For example, facial expressions, gestures, and other non-verbal cues missed during a video conference can result in the need for extended meetings, additional meetings, and / or the need to use additional communication media that require additional consumption of computing resources. Therefore, users may become reluctant to participate in online meetings and video conferences.

[0025] The technical problem raised by video conferencing technology is how to improve the ability of the technology to simulate and / or serve as a viable alternative to face-to-face meetings and conferences. The technical problem arises specifically in the field of the Internet and computer networks, since online meetings can only take place over computer networks, and the problem is rooted in the ability of online meetings and video conferencing technology to enable such meetings.

[0026] To address these technical problems and more, in an example, the present specification provides a technical solution in the form of an eye contact optimization system and method that can be used with video conferencing technology, which can improve the ability of a multimedia conferencing system to simulate a face-to-face meeting by promoting eye contact and back-and-forth conversations between participants in the video conference. The eye contact optimization system is configured to: analyze the audio component and / or video component of a multimedia stream received from a client device to detect characteristics indicating that two or more participants are having a back-and-forth conversation during a multimedia conference, identify the positions of the cameras of the client devices of the participants in the multimedia conference, and control the manner in which the videos of the participants having the conversation are displayed on each other's devices (and in some cases, also on the devices of other participants) so that it appears that the participants having the conversation are making eye contact with each other during the conversation.

[0027] The technical solutions provided herein promote user engagement by promoting verbal and nonverbal communication between participants of online meetings and video conferences. This in turn can lead to improvements in the efficiency of using video conferencing technology, as it can take less time and effort to convey and understand the information transmitted between participants. Users may be reluctant to participate in online meetings and video conferences, and online video conferencing technology may gain even wider acceptance and relevance.

[0028] Figure 1An example multimedia conference system 100 on which aspects of the present disclosure may be implemented is illustrated. The system 100 includes a multimedia conference server 102, a plurality of client devices 106a ... 106n, and a network 109. The system 100 is configured to enable online meetings, video conferences, video calls, multimedia conferences, etc. (hereinafter referred to as "multimedia conferences") to be performed between a plurality of participants 104a ... 104n via a plurality of client devices 106a ... 106n.

[0029] The client devices 106a...106n may be personal or handheld computing devices having or connected to input / output elements that enable users to interact with multimedia conferencing applications such as applications and participate in multimedia conferences. Examples of suitable client devices include, but are not limited to, personal computers, desktop computers, laptop computers, mobile phones, smart phones, tablet computers, tablet phones, smart watches, wearable computers, game consoles, televisions, etc. The client devices 106a...106n are connected to the multimedia conferencing server 102 via a network 109. The network 109 may include one or more wired / wireless communication links and / or communication networks, such as a PAN (personal area network), a LAN (local area network), a WAN (wide area network), or a combination of networks such as the Internet.

[0030] The multimedia conference server 102 is configured to enable multimedia conferences to be scheduled, established, managed, recorded, and controlled via the network. To this end, the multimedia conference server 102 includes a multimedia conference management application 110 that provides this functionality and more via the network. In an embodiment, the multimedia conference server 102 includes one or more computing devices, which may be computers, workstations, servers, server arrays, server farms, etc., and one or more data stores, such as data store 107, which includes data, programmed instructions, and other information that enables multimedia conferencing.

[0031] The client devices 106a...106n include multimedia conference applications 112a...112n that enable participants to connect, schedule, invite other participants, and interact with the multimedia conference management application 110 of the server 102. During a multimedia conference, the multimedia conference application 112a...112n on the client device generates a multimedia stream including live video and audio of the participants. The multimedia conference management application 110 of the server receives the multimedia stream from the client devices 106a...106n and distributes the multimedia conference stream to the client devices 106a...106n via the network 109. The multimedia conference application 112a...112n receives the multimedia conference stream and plays the live video and audio of the participants on the client devices.

[0032] Figure 2 It shows that it can be used to implement Figure 1 106n. The client device 200 includes a display device 210, a speaker 209, a video camera 206, a microphone 207, a network interface 202, a memory 208, and a processor 212. The display device 210 is configured to display text, pictures, graphics, videos, etc. as part of the operation of the client device. The display screen 210 can use any suitable technology, including LCD, plasma, LED, iOLED, CRT, etc. In an embodiment, the display device 210 can be configured as a touch screen display. The speaker 209 is configured to output sounds related to the operation of the client device. The speaker 209 can be integrated into the client device or provided as one or more external components connected to the client device. Any suitable type of speaker can be utilized.

[0033] The video camera 206 is configured to capture video near the client device 200. Any suitable type of video camera can be used. In an embodiment, the video camera 206 can be integrated into the client device, or can be provided as an external component, which can be operably connected to the client device and disconnected from the client device. The video camera 206 is set at a predetermined or pre-selected camera position relative to the display device. In an embodiment, the camera position can be located on or near one of the edges (e.g., top, bottom or side) of the display screen. The microphone 207 is configured to capture audio near the client device 200. The microphone 207 can be integrated into the client device 200 or provided as an external component. Any suitable type of microphone can be used. The network interface 202 enables the client device 200 to connect to a network. The network interface 202 can be configured to have a wired (e.g., Ethernet) and / or wireless (e.g., WIFI) connection to the network.

[0034] The memory 208 can include non-transitory memory, such as random access memory, read-only memory, or a combination thereof. The processor 212 can include a microprocessor, a microcontroller, a digital signal processor, or any combination thereof configured to execute instructions stored in the memory 208. The memory 208 can also store data used with and generated by the execution of the instructions.

[0035] As in Figure 2As shown in , memory 208 stores multimedia conference application 204. Application 204 executed by processor 212 enables users to participate in online multimedia conferences. Examples of applications include, but are not limited to: video conferencing applications (e.g., Microsoft Teams), collaborative work applications, and communication applications. Client device 200 includes at least one user input device 211, such as a keyboard, a pointing device (e.g., a mouse, a trackball, a joystick, etc.), a touch screen, a digitizer, and a stylus, etc., which enables users to interact with the operating system and applications on client device 200.

[0036] During the multimedia conference, the application 204 is configured to control / access the video camera 206 to receive live video and control / access the microphone 207 to receive live audio of the user of the client device 200, and generate a multimedia stream based on the live video and live audio transmitted to the multimedia conference server 102 via the network interface 202 and the network 214. In an embodiment, the application 204 can be configured to generate the multimedia stream in any suitable format and use any suitable and / or necessary protocol that enables the multimedia stream to be properly transmitted and processed by the multimedia conference server. In an embodiment, the application 204 is configured to allow the user to disable the live video and / or audio from the client device, for example, by allowing the user to turn off the camera and microphone.

[0037] The application 204 is also configured to receive a multimedia media conference stream including multimedia streams from client devices of other participants of the multimedia conference from the multimedia media conference server 102. The application 204 includes a stream processing module for processing the multimedia stream and displaying the video from the multimedia stream on the display device 210 and playing the audio from the multimedia stream on the speaker 209. In an embodiment, the application 204 can be configured to display a user interface, such as a window, on the display device in which the video from the multimedia conference is displayed. In an embodiment, the video streams of the participants can be displayed as a gallery, wherein each participant's video stream is shown in a separate frame or sub-window in the user interface, referred to herein as a video tile.

[0038] An example user interface 300 showing a gallery of video tiles 302 is shown in FIG. Figure 3A and Figure 3B In Figure 3A and Figure 3B, the user interface 300 includes a window that can be displayed on the screen of the display device. In an embodiment, the user interface 300 can be displayed so that it occupies the full screen of the display device or only a portion. The size and position of the window 300 can be controlled by the user of the client device, for example, using a user input device. The gallery 302 of video tiles shows six video tiles 304a-304f. Each video tile 304a-304f shows a video stream provided by six corresponding participants of the multimedia conference. The multimedia conference application can control the size and shape of the video tiles.

[0039] The video tiles 304a-304f may be arranged in the gallery in a variety of different ways. Figure 3A In the example of , video tiles 304a-304f are arranged in a grid pattern of two rows, wherein all tiles in the tiles are of the same size. Another example of video tiles are arranged in Figure 3B In Figure 3B , video tiles 304a-304f are arranged so that active participants, e.g., participants associated with tile 304b, are displayed in a central and upper position within window 300 and have a larger tile size, while inactive participants, e.g., 304a, 304c, 304d, 304e, and 304f, are displayed in a row below active participant tile 304b. As used herein, the term "active" as in "active participants" refers to a participant or participants who are actively speaking during the multimedia conference. In contrast, the term "inactive" is used to refer to participants who are not speaking and participants who are not providing live audio (e.g., the audio stream has been turned off or muted).

[0040] An eye contact optimization system that can be integrated into or used in conjunction with the multimedia conference application and / or the multimedia conference management application is provided, which improves the ability of the multimedia conference system to simulate a face-to-face meeting by promoting eye contact and back-and-forth conversation between participants of the multimedia conference. The eye contact optimization system is configured to: analyze the audio component and / or video component of the multimedia stream received from the client device to detect characteristics indicating that two or more participants are engaging in a back-and-forth conversation during the multimedia conference, identify the position of the camera of the client device for the participants in the multimedia conference, and control the manner in which the video of the participants engaging in the conversation is displayed on each other's devices (and in some cases, also on the devices of other participants) so that it appears that the participants engaging in the conversation are engaging in eye contact with each other during the conversation.

[0041] Figure 4An example implementation of an eye contact optimization system 400 according to the present disclosure is shown. The eye contact optimization system 400 includes a session detection module 402, a camera position module 404, and a multimedia stream processing module 406. In an embodiment, these modules can be implemented in a multimedia conference management application on a multimedia conference server 412 or in a multimedia conference application on a client device 414. In some embodiments, the modules can be distributed between the management application and the conference application.

[0042] Figure 4 The eye contact optimization system 400 is configured to receive a multimedia conference stream including a multimedia stream of each of the participants in the multimedia conference from a multimedia conference server 412. The multimedia stream is provided to a multimedia stream processing module 406, which is configured to process the stream to generate a user interface, such as Figure 3A and Figure 3B The user interface 300 includes a gallery of video tiles for display on a display screen 416 of a client device 414, wherein each of the video tiles shows a video stream of a different participant in the conference. The multimedia stream processing module 406 is configured to control the size and arrangement of the video tiles in the gallery. The multimedia stream processing module 404 is also configured to generate a composite audio stream based on the audio stream of the participant provided to the speaker 418 for playback in synchronization with the video stream displayed in the video tile.

[0043] The session detection module 402 is configured to analyze the audio component and / or the video component of the multimedia stream to detect a session between two or more participants and to notify the multimedia stream processing module 406 of the session and the identities of the participants in the session when the session occurs (discussed in more detail below). The camera position module 404 provides the camera position of the client device 414 to the multimedia stream processing module 406. In response to receiving notification that a session between participants has been detected, the multimedia stream processing module 406 can adjust or change the display of the video tiles on the display screen so that the video tiles associated with the participants in the session move toward the camera position on the display screen.

[0044] In an embodiment, the video tile of the participant is moved to a position adjacent to an area of ​​the edge of the display screen closest to the camera. The video tiles of other participants not participating in the session are arranged in a position farther away from the camera than the video tiles of the session participants. In an embodiment, the multimedia stream processing module 406 is configured to control the position of the video tiles within the user interface. In some embodiments, the multimedia stream processing module 406 can be configured to control the size and position of the user interface on the display screen so as to also increase the size of the user interface and / or move the user interface closer to the camera position.

[0045] Figure 5A-Figure 5C Different arrangements of video tiles for different camera positions are shown once a conversation has been detected and the multimedia stream processing module has moved the tiles of the conversation participants closer to the camera position. Figure 5A A display screen 500 is shown with a camera location 502 centered near the top edge of the display screen. A user interface 508 is positioned near the center of the top edge of the display screen, and the video tiles 504a, 504b of the session participants have been moved to positions adjacent to the top edge of the user interface and substantially as close as possible to the user interface 508 and the camera location 502 within the display screen 500. Video tiles 506 of other participants not in the session are positioned in a row near the bottom edge of the interface 508.

[0046] Figure 5B An example is shown in which a display screen 500 has a camera location 502 located near the upper left corner of the display screen 500. In this example, a user interface 508 is positioned in the upper left area of ​​the display screen, and the video tiles 504a, 504b of the session participants have been moved to positions adjacent to the upper left corner of the interface 508 and substantially as close as possible to the user interface 508 and the camera location 502 within the display screen 500. The video tiles 506 of other participants who are not in the session are positioned in a column along the right edge of the interface 508.

[0047] Figure 5C An example is shown in which a display screen 500 has a camera position 502 positioned along the left edge of the display screen. In this example, a user interface 508 is adjacent to the left side of the display screen, and the video tiles 504a, 504b of the session participants have been moved to a position along the left edge of the interface 508 and as close as possible to the user interface 508 and the camera position 502 within the display screen 500.

[0048] In an embodiment, when the system 400 detects that the user of the client device 414 is a participant in a session, the multimedia stream processing module 406 can be configured to move only the tiles associated with the other participant(s) in the session toward the camera position. The multimedia stream processing component 406 can also be configured to increase the size of the video tiles of the other participant(s) on the display screen to further improve the user's ability to see facial expressions, gestures, and other non-verbal cues that can enhance the ability to convey information and promote understanding during the session.

[0049] When the session detection module 402 detects that a session has ended or a new session has started between different groups of participants, the multimedia stream processing module changes the display accordingly. For example, when a session has ended and a new session has not been detected, the video tiles of the participants can be returned to the position and arrangement utilized before the session was detected, such as in Figure 3A and Figure 3B When a new session is detected, the video tiles associated with the participants in the new session move toward the camera location, while participants in previous sessions (who are not participants in the new session) move away from the camera location.

[0050] By moving the video tiles of the participants in the session as close to the camera as possible, when the participants in the session view the video tiles of the other participants in the session, it appears that the participants are looking at each other and making eye contact with each other during the session. Perceived eye contact between session participants and perceived eye contact between other participants and participants in the session promotes user engagement in the video conference, which can improve the efficiency of the video conference, make it easier for participants to understand what is being communicated during the conference, and reduce the amount of time required to convey information during the video conference.

[0051] As mentioned above, the conversation detection module 402 is configured to receive the audio component and / or video component of the multimedia stream of the participant, and process the audio component and / or video component to detect characteristics indicating a conversation (e.g., back-and-forth interaction). In one implementation, the conversation detection module is configured to detect a conversation based on the audio component of the multimedia stream. The conversation detection module is configured to analyze the audio component (e.g., stream) of the participant to identify speech characteristics indicating that two or more participants are actively conversing with each other. An example of a speech characteristic that can be used to detect a conversation is a sequence or pattern of speeches, in which two or more of the same participants speak in an alternating manner (e.g., back-and-forth).

[0052] In an embodiment, a speaker segmentation algorithm may be used to identify sequences and patterns of speech. Speaker segmentation refers to the process of segmenting an input audio stream into homogeneous segments based on speaker identity. A speaker segmentation algorithm may be used to identify which participants have spoken when they are speaking relative to each other. This in turn enables monitoring of the sequence of participants speaking during a conference to detect patterns of speech that indicate a conversation, such as alternating patterns in which two or more participants of the same participant speak after another participant.

[0053] In some embodiments, the conversation detection module 402 may be configured to detect a conversation based on an analysis of the video component of the participant's stream to detect visual characteristics of the participant, which may indicate that a conversation has occurred. For example, the conversation detection module may be configured to use eye tracking technology to analyze the participant's video stream to determine when the participant's gaze appears to switch between the same two locations on the screen in an alternating manner. In an embodiment, eye tracking technology may be used to determine the direction of the participant's gaze, which in turn may be used to determine the location of each participant's display screen. When a participant's gaze appears to switch between two locations on the display screen corresponding to the video tiles of the two participants of the conference, this may be used as an indication that a conversation has occurred between the two participants. In an embodiment, the conversation detection module 402 may be configured to detect a conversation in this manner based on a threshold. For example, the conversation detection module may be configured to detect a conversation when the gaze of a predetermined amount of participants (e.g., all, most, 75%, etc.) appears to switch back and forth between the same two participants.

[0054] In some embodiments, the conversation detection module 402 is configured to analyze the video stream in conjunction with the audio stream of the participant to identify when the conversation occurs, identify the participants in the conversation, and / or confirm the occurrence of the conversation. For example, when the video stream indicates that the participant's gaze is switching between two locations on the display screen, the audio stream can be analyzed to determine which participants are speaking when the participant's gaze is focused on the two locations on the display screen. This information can be used to confirm the occurrence of the conversation and verify the identity of the participants in the conversation.

[0055] An example implementation of the session detection module 650 is shown in FIG. A. The conversation detection module 650 may include a conversation detection model 600, which is configured to receive an audio component and / or a video component of the multimedia stream of the participant as input and is trained to analyze the audio component and / or the video component to identify speech and / or visual features in the audio component and / or the video component that indicate a conversation occurring during the multimedia conference. The conversation detection model 600 is configured to provide an output indicating when a conversation is detected and the participants in the conversation. The conversation detection model 600 is configured to process the audio component and / or the video component to detect a conversation when the audio component and / or the video component is received, so that the conversation is detected in real time when the conversation occurs.

[0056] In an embodiment, the conversation detection module includes a model training component 602, which is configured to train the conversation detection model using training data 606 stored in a training data store 604, thereby providing initial and ongoing training for the conversation detection model 600. In order to train the conversation detection model 600 to detect conversations based on the audio streams of the participants, the conversation detection model 600 is trained to detect and / or recognize human speech and recognize language and / or voice characteristics that indicate a conversation, such as alternating sequences spoken by the same two participants. To this end, the training data for the model 600 includes samples of audio of people conducting conversations. The conversation detection model 600 is trained to learn rules for detecting conversations based on the training data, and applies these rules to the audio streams of the participants in the multimedia conference to detect conversations.

[0057] In order to train the session detection model 600 to detect sessions based on a video stream of participants, the session detection model is trained to identify visual features in the video stream that indicate a session between participants. Visual features may include a participant's gaze switching between two video tiles on a display screen of the client device. To this end, the session detection model is trained to process the video stream using eye tracking technology to monitor where the participant is looking at the display screen. In an embodiment, the training data used to train the model to detect sessions based on a video stream includes samples of videos showing people watching a multimedia conference on a display screen. The session detection model 600 is trained to learn rules for detecting sessions based on the training data, and applies these rules to the video streams of participants in the multimedia conference to detect sessions.

[0058] In an embodiment, the conversation detection model 600 may be trained to analyze the audio stream of the participant in conjunction with the video stream of the participant to detect audio and / or visual features that can be used to detect a conversation and / or confirm the occurrence of a conversation. Alternatively, a separate model may be trained to analyze the audio stream and the video stream to detect a conversation. In an embodiment, the output of the separate model may be provided to another model that is trained to detect a conversation based on a combination of audio and video features identified by the separate model.

[0059] In an embodiment, training data may be derived in part from telemetry data corresponding to user feedback and usage of the session detection module. The telemetry data may be used to enhance learning and / or adjust the training data based on the performance of the model. The telemetry data may be received from a telemetry service 608.

[0060] The session detection model can use any suitable machine learning algorithm (MLA) to detect sessions, including, for example: decision trees, random decision forests, neural networks, deep learning (e.g., convolutional neural networks), support vector machines, regression (e.g., support vector regression, Bayesian linear regression, or Gaussian process regression). The training component can include and / or access a large number of computing resources for training, such as the cloud, including many computer server systems suitable for machine learning training. In some embodiments, the training component is configured to automatically generate multiple different ML models from the same or similar training data for comparison. For example, different underlying MLAs can be trained, such as but not limited to: decision trees, random decision forests, neural networks, deep learning (e.g., convolutional neural networks), support vector machines, regression (e.g., support vector regression, Bayesian linear regression, or Gaussian process regression). As another example, the size or complexity of the model can vary between different ML models, such as for the maximum depth of a decision tree, or the number and / or size of hidden layers in a convolutional neural network. Furthermore, different training methods may be used to train different ML models, such as, but not limited to, the selection of training, validation, and testing data, the ordering and / or weighting of training data items, or the number of training iterations. One or more of the resulting multiple trained ML models may be selected based on factors such as, but not limited to, accuracy, computational efficiency, and / or power efficiency. In some implementations, a single trained ML model may be generated.

[0061] In an embodiment, when the conversation detection model 600 detects a conversation, the model generates an output that notifies the multimedia stream processing module 406 that a conversation is occurring and indicates the participants in the conversation. The multimedia stream processing module 406 then changes the display of the video tiles as described above so that the video tiles associated with the participants in the conversation move on the display screen toward the camera position.

[0062] In an embodiment, the session detection module 650 may be configured to exclude participants from consideration because their session does not provide audio and / or video for the multimedia conference. For example, a participant who mutes the audio and / or disables the camera of their device may be used as an indication that the user is not participating in the session. This is particularly true when a participant does not provide audio indicating that the participant does not want to participate in the session or is unable to participate (e.g., does not have a microphone). The session detection module 402 may exclude the feeds of these participants from being provided as input to the session detection model 600. In some cases, based on the audio stream from the participant, a participant who does not provide video may still be considered an active participant in the session. Even if eye contact optimization will not benefit the participant, the video tile associated with the participant (in this case, it will typically display a graphic, such as an icon and / or initial item, replacing the participant's video stream) may still be moved toward the camera position to indicate which of the participants are currently in session.

[0063] In some embodiments, the conversation detection model 600 is also trained to detect when a conversation ends. In an embodiment, the end of a conversation can be detected when one or both of the participants in the conversation do not speak for a predetermined amount of time, or when a new conversation between different participants is detected before the end of a previous conversation. In response to detecting the end of a conversation, the conversation detection model 600 generates an output notifying the multimedia stream processing module that the conversation has ended. As discussed above, the multimedia stream processing module can then return the video tiles to the positioning and arrangement utilized before the conversation was detected. When a new conversation has been detected before the end of a previous conversation, the multimedia stream processing module is configured to replace the video tiles of the participants of the previous conversation with the video tiles of the participants of the new conversation. The video tiles of the previous participants can be returned to their initial positions on the display screen.

[0064] Reference again Figure 4, the camera position module 404 is used to provide the camera position of the client device 414 to the multimedia stream processing module 406. The camera position module 404 can be configured to identify the camera position in a variety of ways. In embodiments where the video camera is integrated into the client device as a built-in camera, the camera position module 404 can be configured to identify the camera position by requesting the camera position information from the client device 414. In some embodiments, the camera position module 404 is configured to determine the type and / or model of the client device and access specifications for the type and / or model to determine the camera position. In an embodiment, the specifications can be obtained from a memory that can be accessed by the camera position module 404 or obtained via a network from a remote location such as a website or other data repository.

[0065] exist Figure 6B An example implementation of a camera position module for automatically detecting the camera position and / or orientation of a camera on a client device is shown in . In an embodiment, the camera position module 660 may include a camera detection model 662, which is configured to receive a video component of a multimedia stream of a participant as input, and is trained to analyze the video component using one or more of an eye tracking algorithm, a face tracking algorithm, a head tracking algorithm, a head pose estimation algorithm, a gaze estimation algorithm, etc. to detect a visual characteristic of the participant indicating that the participant is viewing the display screen, and specifically, at the center of the display screen, and estimate the camera position and / or orientation based on the detected visual characteristics. As an example, the camera detection model may be configured to track the head orientation and / or gaze direction of the participant over time, and determine an average head orientation and / or gaze direction, which can be used as an estimate of the head orientation when viewing the central area of ​​the display screen. Then, the camera position / orientation can be estimated based on the determined head orientation and / or gaze direction.

[0066] Figure 6C An illustration of faces 680, 682, 684, 686 relative to the view / camera angle is shown. Face 680 is shown looking straight ahead, i.e., looking directly at the camera. Face 682 is oriented in direction 688 downward and to the right relative to the view / camera angle. Face 684 is oriented in direction 690 downward and centered relative to the view / camera angle. Face 686 is oriented in direction 692 downward and to the left relative to the view / camera angle. Known algorithms and signal processing techniques are capable of estimating head orientation and / or gaze direction from video images. Detecting head orientation and / or gaze direction in turn enables estimation of camera position / orientation. For example, in Figure 6C, camera positions / orientations 694, 696, 698 are substantially opposite to directions 688, 690, 692, respectively, i.e., the face is oriented relative to the view / camera angle. In an embodiment, the camera detection model 662 may be trained to analyze the video of the participant and estimate the head orientation and / or gaze direction indicating that the participant is viewing the display screen, then estimate the camera position / orientation based on the estimated head orientation and / or gaze direction, and provide the camera position / orientation as an output.

[0067] refer to Figure 6B , the camera detection model 662 includes a model training component 664, which is configured to train the camera detection model 662 using training data 666 stored in a training data store 668 to provide initial and ongoing training for the camera detection model 662. In order to train the camera detection model 662 to estimate the camera position / orientation based on the video streams of the participants, the camera detection model 662 is trained to detect faces and recognize language and / or speech characteristics indicative of a conversation, such as alternating sequences spoken by the same two participants. To this end, the training data for the model 662 includes audio samples of people conducting a conversation. The camera detection model 662 is trained to learn rules for estimating head orientation and / or gaze direction, which indicates that the participant is looking at a display screen, and then, determine the camera position / orientation based on the estimated head orientation and / or gaze direction.

[0068] In an embodiment, the camera position module 404 may be configured to implement a calibration process to determine the camera position for a client device. This is particularly useful in the case where a video camera is provided as a separate external component for the client device. The calibration process involves requesting and receiving some input and / or interaction from a user associated with the client device, which can be used to directly or indirectly determine the camera position for the client device. As an example, the calibration process may involve displaying a visual indicator on a display screen, such as an image of a video tile or an object similar to a video tile, and prompting the user to move the visual indicator toward the camera using a user input device such as a mouse. As another example, the calibration process may prompt the user to view the visual indicator when the visual indicator moves in a direction that causes the user to look at the video camera. In either case, the visual indicator may move toward the camera to the extent possible within the display screen. The final position of the visual indicator may then be used as an approximation of the camera position for the device. As another example, the calibration process may involve displaying in an image representing the display screen (e.g., such as a square or rectangle), and prompting the user to select a position (e.g., using a mouse) relative to the image of the display screen where the camera is located.

[0069] In some implementations, the camera position module 404 may be configured to automatically determine / estimate the camera position for the client device without prompting the user for a particular type of input / interaction. In an embodiment, the camera position module may be configured to determine the camera position by analyzing a video image of a user of the client device to detect visual characteristics indicating that the user is looking at the camera. As an example, the camera position module 404 may be configured to detect visual characteristics indicating that the user's face is oriented directly toward the camera based on the position of features such as the shape of the face and / or features of the face (such as eyes, nose, and mouth) related to the overall shape of the face. As another example, the camera position module 404 may be configured to detect visual characteristics of an eye focused on the camera, such as an iris and / or pupil being round, wherein the pupil is centered in the iris.

[0070] Figure 7 700 is a flow chart of an example method 700 for conducting a video conference with multiple participants using an eye contact optimization system according to the present disclosure. The method 700 begins by receiving a multimedia stream from a client device associated with a participant in the video conference (block 702). The multimedia stream includes both audio and video components. The multimedia stream processing module generates a gallery of video tiles based on the video component of the multimedia stream displayed on a display screen of the client device (block 704). Each of the video tiles is used to display a video component from one of the participants in the video conference. A camera position for a camera of the client device is determined, for example, by a camera position module (block 706).

[0071] The audio component and / or video component of the multimedia stream is analyzed by a session detection module to detect characteristics indicating that the first participant and the second participant are conducting a session (box 708). In response to detecting a session between the first participant and the second participant, the display of the video tiles associated with the first participant and the second participant is moved toward the camera position on the display screen of the client device, and moved to a position where the video tiles are adjacent to the edge of the display screen where the camera position is located (box 710). Then, control returns to box 708 because the audio component and / or video component of the multimedia stream continues to be monitored to detect a session. The stream is also monitored to detect the end of a session (box 712). When the end of the current session is detected without detecting a new session, the video tiles of the first participant and the second participant can be returned to their initial positions on the display screen (box 714). When a new session is detected before the end of the current session is detected, the video tiles of the participants of the new session can be moved toward the camera position to a position where the video tiles are adjacent to the edge of the display screen where the camera position is located, and the video tiles of the participants of the previous session can be returned to their previous positions on the display screen.

[0072] Figure 8 is a block diagram 800 illustrating an example software architecture 802, portions of which may be used in conjunction with the various hardware architectures described herein that may implement any of the features described above. Figure 8 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 802 may be implemented in a system such as Fig. 9 900, which includes, among other things, a processor 910, a memory 930, and input / output (I / O) components 950. Representative hardware layers 804 are illustrated and can represent, for example, Fig. 9 The machine 900 of the embodiment of the present invention. The representative hardware layer 804 includes a processing unit 806 and associated executable instructions 808. The executable instructions 808 represent executable instructions of the software architecture 802, including implementations of the methods, modules, etc. described herein. The hardware layer 804 also includes a memory / storage device 810, which also includes executable instructions 808 and accompanying data. The hardware layer 804 may also include other hardware modules 812. The instructions 808 held by the processing unit 806 may be part of the instructions 808 held by the memory / storage device 810.

[0073] The example software architecture 802 can be conceptualized as layers, each providing various functions. For example, the software architecture 802 may include layers and components such as an operating system (OS) 814, a library 816, a framework 818, an application 820, and a presentation layer 844. In operation, applications 820 and / or other components within a layer may invoke API calls 824 to other layers and receive corresponding results 826. The illustrated layers are representative in nature, and other software architectures may include additional or different layers. For example, some mobile or dedicated operating systems may not provide a framework / middleware 818.

[0074] OS 814 can manage hardware resources and provide public services. OS 814 may include, for example, a kernel 828, a service 830, and a driver 832. Kernel 828 may serve as an abstraction layer between hardware layer 804 and other software layers. For example, kernel 828 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Service 830 may provide other public services for other software layers. Driver 832 may be responsible for controlling or docking with underlying hardware layer 804. For example, depending on hardware and / or software configuration, driver 832 may include a display driver, a camera driver, a memory / storage device driver, a peripheral device driver (e.g., via a universal serial bus (USB)), a network and / or wireless communication driver, an audio driver, etc.

[0075] The library 816 may provide a common infrastructure that may be used by the application 820 and / or other components and / or layers. The library 816 generally provides functions used by other software modules to perform tasks, rather than directly interacting with the OS 814. The library 816 may include a system library 834 (e.g., a C standard library) that may provide functions such as memory allocation, string manipulation, and file operations. In addition, the library 816 may include an API library 836, such as a media library (e.g., supporting the presentation and manipulation of image, sound, and / or video data formats), a graphics library (e.g., an OpenGL library for rendering 2D and 3D graphics on a display), a database library (e.g., SQLite or other relational database functions), and a web library (e.g., WebKit that may provide web browsing functions). The library 816 may also include a variety of other libraries 838 to provide many functions for the application 820 and other software modules.

[0076] Framework 818 (sometimes also referred to as middleware) provides a higher level common infrastructure that can be used by applications 820 and / or other software modules. For example, framework 818 can provide various graphical user interface (GUI) functions, advanced resource management, or advanced location services. Framework 818 can provide a wide range of other APIs for applications 820 and / or other software modules.

[0077] Applications 820 include built-in applications 840 and / or third-party applications 842. Examples of built-in applications 840 may include, but are not limited to, contact applications, browser applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 842 may include any application developed by an entity other than the vendor of a particular platform. Applications 820 may use the functionality available via OS 814, library 816, framework 818, and presentation layer 844 to create a user interface to interact with a user.

[0078] Some software architectures use virtual machines, as illustrated by virtual machine 848. Virtual machine 848 provides an execution environment in which applications / modules can run as they would on a hardware machine (e.g., such as a Fig. 9 The virtual machine 848 may be hosted by a host OS (e.g., OS 814) or a hypervisor and may have a virtual machine monitor 846 that manages the operation of the virtual machine 848 and interoperates with the host operating system. A software architecture that may be different from the software architecture 802 external to the virtual machine is executed within the virtual machine 848, such as an OS 850, a library 852, a framework 854, an application 856, and / or a presentation layer 858.

[0079] Fig. 9 900 is a block diagram illustrating components of an example machine 900 configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any of the features described herein. The example machine 900 is in the form of a computer system within which instructions 916 (e.g., in the form of software components) for causing the machine 900 to perform any of the features described herein can be executed. In this way, the instructions 916 can be used to implement the modules or components described herein. The instructions 916 cause the unprogrammed and / or unconfigured machine 900 to operate as a specific machine configured to perform the described features. The machine 900 can be configured to operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a node in a peer-to-peer or distributed network environment. The machine 900 may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smart phone, a mobile device, a wearable device (e.g., a smart watch), and an Internet of Things (IoT) device. In addition, although only a single machine 900 is illustrated, the term "machine" includes a collection of machines that individually or jointly execute instructions 916.

[0080] The machine 900 may include a processor 910, a memory 930, and I / O components 950, which may be communicatively coupled via, for example, a bus 902. The bus 902 may include multiple buses that couple various elements of the machine 900 via various bus technologies and protocols. In an example, the processor 910 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 912a to 912n that can execute instructions 916 and process data. In some examples, one or more processors 910 may execute instructions provided or identified by the one or more processors 910. The term "processor" includes a multi-core processor, which includes cores that can execute instructions simultaneously. Although Fig. 9 Multiple processors are shown, but the machine 900 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine 900 may include multiple processors distributed among multiple machines.

[0081] The memory / storage 930 may include a main memory 932, a static memory 934, or other memory and storage unit 936, both of which can be accessed by the processor 910, such as via the bus 902. The storage unit 936 and the memory 932 store instructions 916 that embody any one or more of the functions described herein. The memory / storage 930 may also store temporary data, intermediate data, and / or long-term data for the processor 910. The instructions 916 may also reside, in whole or in part, within the memory 932, 934, within the storage unit 936, within at least one of the processors 910 (e.g., within a command buffer or cache memory), within a memory within at least one of the I / O components 950, or any suitable combination thereof. Therefore, the memory 932, 934, the storage unit 936, the memory in the processor 910, and the memory in the I / O component 950 are examples of machine-readable media.

[0082] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions and data that cause the machine 900 to operate in a particular manner, and may include, but is not limited to: random access memory (RAM), read-only memory (ROM), cache memory, flash memory, optical storage media, magnetic storage media and devices, cache memory, network accessible or cloud storage, other types of storage devices, and / or any suitable combination thereof. The term "machine-readable medium" applies to a single medium or a combination of multiple media used to store instructions (e.g., instructions 916) for execution by the machine 900, such that the instructions, when executed by one or more processors 910 of the machine 900, cause the machine 900 to perform one or more of the features described herein. Thus, a "machine-readable medium" may refer to a single storage device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" excludes the signal itself.

[0083] I / O components 950 may include a wide variety of hardware components suitable for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 950 included in a particular machine will depend on the type and / or function of the machine. For example, a mobile device such as a mobile phone may include a touch input device, while a headless server or IoT device may not include such a touch input device. Fig. 9 The specific examples of I / O components illustrated in the figure are by no means limiting, and other types of components may be included in the machine 900. The grouping of the I / O components 950 is only used to simplify the discussion, and the grouping is by no means limiting. In various examples, the I / O components 950 may include a user output component 952 and a user input component 954. The user output component 952 may include, for example: a display component (e.g., a liquid crystal display (LCD) or a projector) for displaying information, an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor or a force feedback device), and / or other signal generators. The user input component 954 may include, for example: an alphanumeric input component (e.g., a keyboard or a touch screen), a pointing component (e.g., a mouse device, a touch pad, or another pointing instrument), and / or a tactile input component (e.g., a physical button or touch screen that provides the location and / or force of a touch or touch gesture), which is configured to receive various user inputs, such as user commands and / or selections.

[0084] In some examples, I / O components 950 may include biometric components 956, motion components 958, environment components 960, and / or position components 962, as well as various other physical sensor components. Biometric components 956 may include, for example, components for detecting body expressions (e.g., facial expressions, vocal expressions, hand or body gestures, or eye tracking), measuring biosignals (e.g., heart rate or brain waves), and identifying people (e.g., via voice-based, retinal, fingerprint, and / or face-based recognition). Motion components 958 may include, for example, acceleration sensors (e.g., accelerometers) and rotation sensors (e.g., gyroscopes). Environment components 960 may include, for example, lighting sensors, temperature sensors, humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., microphones for detecting ambient noise), proximity sensors (e.g., infrared sensing of nearby objects), and / or other components that may provide indications, measurements, or signals corresponding to the surrounding physical environment. Position component 962 may include, for example, a position sensor (e.g., a global positioning system (GPS) receiver), an altitude sensor (e.g., a barometric pressure sensor from which altitude can be derived), and / or an orientation sensor (e.g., a magnetometer).

[0085] The I / O components 950 may include a communication component 964 that implements various techniques operable to couple the machine 900 to a network 970 and / or a device 980 via respective communication couplings 972 and 982. The communication component 964 may include one or more network interface components or other suitable devices to interface with the network 970. The communication component 964 may include, for example, components suitable for providing wired communication, wireless communication, cellular communication, near field communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device 980 may include other machines or various peripheral devices (e.g., via USB coupling).

[0086] In some examples, the communication component 964 can detect the identifier or include a component suitable for detecting the identifier. For example, the communication component 964 can include a radio frequency identification (RFID) tag reader, an NFC detector, an optical sensor (e.g., a one-dimensional or multi-dimensional barcode or other optical code), and / or an acoustic detector (e.g., a microphone for identifying an audio signal of a tag). In some examples, the location information can be determined based on information from the communication component 962 (such as, but not limited to, a geographic location via an Internet Protocol (IP) address, a geographic location via Wi-Fi, cellular, NFC, Bluetooth, or other wireless station identification and / or signal triangulation).

[0087] In the following, further features, characteristics and advantages of the present invention will be described by the following items:

[0088] Item 1. A method for conducting a video conference with a plurality of participants, each of the participants utilizing a device for transmitting a multimedia feed of the participant associated with the device and receiving multimedia feeds from devices associated with other participants in the video conference, the method comprising:

[0089] receiving the multimedia feed from the plurality of devices associated with each of the participants in the video conference, the multimedia feed comprising an audio component and a video component;

[0090] displaying tiles on a display screen of the device of each of the participants, each of the tiles respectively showing the video component of one of the multimedia feeds;

[0091] analyzing at least one of the audio component and the video component of a plurality of multimedia feeds to detect a characteristic indicating that a first participant and a second participant are engaging in a first conversation with each other;

[0092] identifying a first camera position for a camera on a device of the first participant and a second camera position for a camera on a device of the second participant;

[0093] In response to identifying that the first participant and the second participant are having the first conversation with each other:

[0094] moving a tile for the first participant on a display screen of the device of the second participant from a first position to a second position, the second position being closer to the second camera position than the first position; and

[0095] The tile for the second participant on the display screen of the device of the first participant is moved from a third position to a fourth position, the fourth position being closer to the first camera position than the third position.

[0096] Item 2. The method of item 1, wherein analyzing the at least one of the audio component and the video component of each of the multimedia feeds further comprises:

[0097] The video component of the multimedia feed is analyzed to detect characteristics indicating that the participants are viewing a tile on a display screen of their respective devices.

[0098] Item 3. The method according to any one of items 1-2, further comprising:

[0099] detecting when the video component of the multimedia feed indicates that the participant's gaze is primarily switching between a first tile and a second tile on a display screen of their respective device; and

[0100] The first participant is identified as a participant associated with the first tile, and the second participant is identified as a participant associated with the second tile.

[0101] Item 4. The method according to any one of items 1 to 3, wherein the detection further comprises:

[0102] determining a tile position for each of the tiles on each of the display devices of the participants; and

[0103] The video component of the multimedia feed is analyzed to identify which of the tile locations a respective participant is looking at.

[0104] Item 5. The method of any one of items 1-4, wherein analyzing the at least one of the audio component or the video component of each of the multimedia feeds further comprises:

[0105] analyzing the audio component of the multimedia feed to detect speech characteristics indicative of two participants conducting the conversation;

[0106] The two participants are identified as the first participant and the second participant based on the speech characteristics.

[0107] Item 6. A method according to any one of items 1-5, wherein the speech characteristics include the two participants speaking in an alternating manner.

[0108] Item 7. The method according to any one of items 1 to 6, further comprising:

[0109] analyzing at least one of the audio component and the video component of each of the multimedia feeds to detect a characteristic indicating that the first participant and a third participant conducted a second conversation subsequent to the first conversation;

[0110] identifying a third camera location for a camera on a device of the third participant;

[0111] In response to identifying that the first participant and the third participant are conducting the second conversation:

[0112] moving the tile for the first participant on the display screen of the third participant's device from a fifth position to a sixth position, the sixth position being closer to the third camera position than the fifth position; and

[0113] The tile of the second participant at the second position is replaced with the tile for the third participant.

[0114] Item 8. The method according to any one of items 1 to 7, wherein identifying the first camera position further comprises:

[0115] analyzing the video component of the multimedia feed of the first participant using at least one of an eye tracking algorithm, a face tracking algorithm, and a head tracking algorithm to determine a head orientation of the first participant indicating that the first participant is viewing the display screen; and

[0116] The first camera position is estimated based on the head orientation.

[0117] Item 9. The method according to any one of items 1-8, wherein identifying the first camera position further comprises:

[0118] analyzing the video component of the multimedia feed of the first participant using at least one of an eye tracking algorithm, a face tracking algorithm, and a head tracking algorithm to identify when the first participant is looking at the display screen and when the first participant is looking at the camera;

[0119] determining a first gaze direction when the first participant is viewing the display screen and a second gaze direction when the first participant is viewing the camera; and

[0120] The first camera position is estimated based on a comparison of the first gaze direction and the second gaze direction.

[0121] Item 10. The method of any one of clauses 1-9, wherein identifying the first camera position further comprises:

[0122] displaying a visual representation of the display screen; and

[0123] receiving user input indicating a position of the camera relative to the visual representation; and

[0124] Set the position to be the first camera position.

[0125] Item 11. The method according to any one of items 1-10, wherein identifying the first camera position further comprises:

[0126] displaying a camera position indication on the display screen;

[0127] receiving a user input that causes the camera position indicator to move from a first position to a second position that is closer to the camera than the first position; and

[0128] The second position is set as the first camera position.

[0129] Item 12. A data processing system comprising:

[0130] processor;

[0131] A memory in communication with the processor, the memory including executable instructions that, when executed by the processor, cause the data processing system to perform the following functions:

[0132] Receiving multimedia streams of a plurality of participants in a multimedia conference, the multimedia streams comprising an audio component and a video component;

[0133] displaying video tiles on a display screen of a device of at least one of the participants, each of the video tiles showing the video component of one of the multimedia streams;

[0134] analyzing the audio component and / or the video component of the multimedia stream to detect characteristics indicating that a first participant and a second participant are engaging in a first conversation with each other;

[0135] identifying a camera position for a camera of the device; and

[0136] In response to identifying that the first participant and the second participant are conducting the first conversation with each other, a video tile for the first participant and a video tile for the second participant are moved to an edge of the display screen in the direction of the camera position.

[0137] Item 13. The data processing system of Item 12, wherein analyzing the audio component and / or the video component of each of the multimedia feeds further comprises:

[0138] The video component of the multimedia feed is analyzed to detect visual characteristics indicative of participants conducting a conversation.

[0139] Item 14. The data processing system according to any one of Items 12-13, wherein the function further comprises:

[0140] analyzing the video component to detect when a participant's gaze switches between a location of a first video tile and a location of a second video tile; and

[0141] The first participant is identified as a participant associated with the first video tile, and the second participant is identified as a participant associated with the second video tile.

[0142] Item 15. The data processing system according to any one of Items 12-14, wherein analyzing the video component to detect the gaze of the participant further comprises:

[0143] using an eye tracking method to determine the gaze direction of the participant in the video component of the multimedia stream; and

[0144] The first video tile and the second video tile are identified based on the gaze direction of the participant.

[0145] Item 16. The data processing system according to any one of Items 12-15, wherein analyzing the audio component and / or the video component of the multimedia stream further comprises:

[0146] analyzing the audio component of the multimedia stream to detect speech characteristics indicative of two participants conducting the conversation;

[0147] The two participants are identified as the first participant and the second participant based on the speech characteristics.

[0148] Item 17. A data processing system according to any one of Items 12-16, wherein the speech characteristics include the two participants speaking in an alternating manner.

[0149] Item 18. A non-transitory computer-readable medium having stored thereon instructions that, when executed, cause a programmable device to perform the following functions:

[0150] Receiving multimedia streams of a plurality of participants in a multimedia conference, the multimedia streams comprising an audio component and a video component;

[0151] displaying video tiles on a display screen of a device of at least one of the participants, each of the video tiles showing the video component of a video tile in the multimedia stream;

[0152] analyzing the audio component and / or the video component of the multimedia stream to detect characteristics indicating that a first participant and a second participant are engaging in a first conversation with each other;

[0153] identifying a camera position for a camera of the device;

[0154] In response to identifying that the first participant and the second participant are having the first conversation with each other:

[0155] The video tile for the first participant and the video tile for the second participant are moved to an edge of the display screen in the direction of the camera position.

[0156] Item 19. The non-transitory computer-readable medium of Item 18, wherein the functions further include:

[0157] analyzing the audio component and / or the video component of the multimedia stream to detect that the first participant and a third participant conduct a second conversation after the first conversation;

[0158] In response to identifying that the first participant and the third participant are conducting the second conversation:

[0159] replacing the second video tile of the second participant with the third video tile of the third participant; and

[0160] The second video tile is returned to a previous position on the display screen.

[0161] Item 20. The non-transitory computer-readable medium of any one of Items 18-19, wherein identifying the camera position further comprises:

[0162] A calibration process is performed on the device, wherein a user of the device is prompted to provide input indicative of a position of the camera.

[0163] Although various embodiments have been described, the description is intended to be exemplary rather than restrictive, and it should be understood that within the scope of the embodiments, many more embodiments and implementations are possible. Although many possible combinations of features are shown in the drawings and discussed in this detailed description, many other combinations of the disclosed features are possible. Unless specifically limited, any feature of any embodiment may be used or substituted in combination with any other feature or element in any other embodiment. Therefore, it should be understood that any feature shown and / or discussed in this disclosure may be implemented together in any suitable combination. Therefore, the embodiments are not limited except for the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of the appended claims.

[0164] Although the foregoing has described what is considered to be the best mode and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which are described herein. The appended claims are intended to include any and all applications, modifications, and variations that fall within the true scope of the present teachings.

[0165] Unless otherwise indicated, all measurements, values, ratings, positions, sizes, dimensions and other specifications set forth in this specification (including in the following claims) are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they relate and customary in the art to which they belong.

[0166] The scope of protection is limited only by the following claims. The scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims when interpreted in light of this specification and the prosecution history that follows, and to encompass all structural and functional equivalents. Nevertheless, none of the claims is intended to encompass subject matter that fails to meet the requirements of sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintended envelopment of such subject matter is hereby claimed.

[0167] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as causing exclusive use to the public of any component, step, feature, object, benefit, advantage, or equivalent, whether or not recited in the claims.

[0168] It will be understood that the terms and expressions used in this article have the common meaning consistent with such terms and expressions about their corresponding inquiries and research fields, unless a specific meaning is set forth in this article in addition. Relational terms such as first and second etc. can be used only to distinguish an entity or action from another entity or action, without requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "include" or any other variation thereof are intended to cover non-exclusive inclusions, so that the process, method, product or device including the list of elements does not only include those elements, but may include other elements that are not explicitly listed or inherent to such processes, methods, products or devices. The elements preceded by "one" or "an" do not exclude the presence of other identical elements in the process, method, product or device including the element without further constraints.

[0169] An abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It should be understood that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that various features are grouped together in various examples for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claims require more features than those expressly recited in each claim. On the contrary, as reflected in the following claims, the subject matter of the present invention lies in less than all the features of a single disclosed example. Therefore, the following claims are incorporated into the specific implementation, with each claim itself as a separately claimed subject matter.

Claims

1. A method for conducting a video conference with a plurality of participants, each of the participants utilizing a device for transmitting a multimedia feed of the participant associated with the device and receiving multimedia feeds from devices associated with other participants in the video conference, the method comprising: receiving the multimedia feed from a plurality of devices associated with each of the participants in the video conference, the multimedia feed comprising an audio component and a video component; displaying tiles on a display screen of the device of each of the participants, each of the tiles respectively showing the video component of one of the multimedia feeds; analyzing at least one of the audio component and the video component of a plurality of the multimedia feeds to detect a characteristic indicative of a first participant and a second participant engaging in a first conversation with each other; identifying a first camera position for a camera on a device of the first participant and a second camera position for a camera on a device of the second participant; In response to identifying that the first participant and the second participant are having the first conversation with each other: moving a tile for the first participant on a display screen of a device of the second participant from a first position to a second position, the second position being closer to the second camera position than the first position; as well as The tile for the second participant is moved on a display screen of the first participant's device from a third position to a fourth position, the fourth position being closer to the first camera position than the third position.

2. The method according to claim 1, wherein: Analyzing the at least one of the audio component and the video component of each of the multimedia feeds further comprises: The video component of the multimedia feed is analyzed to detect characteristics indicating that the participants are viewing a tile on a display screen of their respective devices.

3. The method according to claim 2, further comprising: detecting when the video component of the multimedia feed indicates that the participant's gaze is primarily switching between a first tile and a second tile on a display screen of their respective device; as well as The first participant is identified as a participant associated with the first tile, and the second participant is identified as a participant associated with the second tile.

4. The method according to claim 3, wherein: The detection also includes: determining a tile position for each of the tiles on each of the display devices of the participants; and The video component of the multimedia feed is analyzed to identify which of the tile locations a respective participant is viewing.

5. The method according to claim 1, wherein: Analyzing the at least one of the audio component or the video component of each of the multimedia feeds further comprises: analyzing the audio component of the multimedia feed to detect speech characteristics indicative of two participants conducting the conversation; The two participants are identified as the first participant and the second participant based on the speech characteristics.

6. The method according to claim 5, wherein: The speech characteristics include the two participants speaking in an alternating manner.

7. The method according to claim 1, further comprising: analyzing the at least one of the audio component and the video component of each of the multimedia feeds to detect a characteristic indicating that the first participant conducted a second conversation with a third participant subsequent to the first conversation; identifying a third camera location for a camera on a device of the third participant; In response to identifying that the first participant and the third participant are conducting the second conversation: moving the tile for the first participant on the display screen of the third participant's device from a fifth position to a sixth position, the sixth position being closer to the third camera position than the fifth position; and The tile of the second participant at the second position is replaced with the tile for the third participant.

8. The method according to claim 1, wherein: Identifying the first camera position further includes: analyzing the video component of the multimedia feed of the first participant using at least one of an eye tracking algorithm, a face tracking algorithm, and a head tracking algorithm to determine a head orientation of the first participant indicating that the first participant is viewing the display screen; and The first camera position is estimated based on the head orientation.

9. The method according to claim 1, wherein: Identifying the first camera position further includes: analyzing the video component of the multimedia feed of the first participant using at least one of an eye tracking algorithm, a face tracking algorithm, and a head tracking algorithm to identify when the first participant is looking at the display screen and when the first participant is looking at the camera; determining a first gaze direction when the first participant is viewing the display screen and a second gaze direction when the first participant is viewing the camera; and The first camera position is estimated based on a comparison of the first gaze direction and the second gaze direction.

10. The method according to claim 1, wherein: Identifying the first camera position further includes: displaying a visual representation of the display screen; and receiving user input indicating a position of the camera relative to the visual representation; and Set the position to be the first camera position.

11. The method according to claim 1, wherein: Identifying the first camera position further includes: displaying a camera position indicator on the display screen; receiving a user input that causes the camera position indicator to move from a first position to a second position that is closer to the camera than the first position; and The second position is set as the first camera position.

12. A data processing system comprising: processor; A memory in communication with the processor, the memory including executable instructions that, when executed by the processor, cause the data processing system to perform the following functions: Receiving multimedia streams of a plurality of participants in a multimedia conference, the multimedia streams comprising an audio component and a video component; displaying video tiles on a display screen of a device of at least one of the participants, each of the video tiles showing the video component of one of the multimedia streams; analyzing the audio component and / or the video component of the multimedia stream to detect characteristics indicating that a first participant and a second participant are engaging in a first conversation with each other; identifying a camera position for a camera of the device; as well as In response to identifying that the first participant and the second participant are conducting the first conversation with each other, a video tile for the first participant and a video tile for the second participant are moved to an edge of the display screen in the direction of the camera position.

13. The data processing system according to claim 12, wherein: Analyzing the audio component and / or the video component of each of the multimedia feeds further comprises: The video component of the multimedia feed is analyzed to detect visual characteristics indicative of participants conducting a conversation.

14. The data processing system according to claim 13, wherein: The functionality also includes: analyzing the video component to detect when a participant's gaze switches between a location of a first video tile and a location of a second video tile; and The first participant is identified as a participant associated with the first video tile, and the second participant is identified as a participant associated with the second video tile.

15. The data processing system according to claim 14, wherein: Analyzing the video component to detect the participant's gaze further includes: using an eye tracking method to determine the gaze direction of the participant in the video component of the multimedia stream; and The first video tile and the second video tile are identified based on the gaze direction of the participant.

Images