Eye contact assistance in video conferencing

Abstract

Generally discussed herein are devices, systems, and methods for eye contact assistance in video conferences. A method may include determining a position of a web camera relative to a display; providing a user interface of the video conference on the display; and providing a graphical display of information on the user interface at a pixel of the display closest to the location.

Description

Background Art

[0001] Eye contact can be important in interpersonal communication. Eye contact helps people build trust, establish relationships, persuade, etc. Studies on face-to-face meetings show that people tend to make eye contact up to about 60% of the time during face-to-face meetings. Maintaining eye contact in online meetings can be challenging. Summary of the invention

[0002] A device, system, method, and computer-readable medium configured for eye contact assistance in video conferencing. Eye contact can be supported by attracting the gaze of the participant close to the camera. Attracting gaze can include displaying information at the pixel of the display closest to the camera or changing the information displayed at the pixel of the display closest to the camera. The information presentation can be customized by the participant, such as changing the form or content of the information presented closest to the display. The conditions for presenting the information closest to the display can be defined by the participant. Other innovations can be achieved through the teachings of this specification.

[0003] A method may include determining a position of a web camera relative to a display. The method may include providing a user interface for a video conference on the display. A graphical display of information may be provided on the user interface at a pixel of the display closest to the position. Video data may be received from the web camera. A gaze direction of a user in the video data may be determined based on the video data. The position of the web camera may be inferred based on the gaze direction.

[0004] Determining the position of the web camera may include prompting a user to provide the position by a user interface; and receiving the position from the user. Trigger data may be defined by the user. The trigger data may be analyzed to determine that a condition is satisfied. In response to determining that the condition is satisfied, information may be provided by the pixel.

[0005] The method may also include receiving, by the user interface, information data indicating the content of the information from the user. The method may also include receiving, by the user interface, form data indicating how the content is to be presented from the user. The method may also include presenting, by the user interface, the content in the form indicated by the form data. The information may include video data of active speakers, content of a chat window, a shared screen. The information may include data received by a second application, the second application being different from the application providing the video conferencing functionality.

[0006] The method may also include determining that the user is not looking at the camera, and in response to determining that the user is not looking at the camera, providing information at the pixel. Determining that the user is not looking at the camera includes determining that the user has not looked at the camera for a specified amount of time. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A block diagram of an embodiment of a video conferencing system is illustrated by way of example.

[0008] Figure 2 A block diagram of an embodiment of a system for eye contact assistance in video conferencing is illustrated by way of example.

[0009] Figure 3 A perspective view of an embodiment in which participants make insufficient eye contact is illustrated by way of example.

[0010] Figure 4 A block diagram of an embodiment of the system after an object has been moved into proximity with a camera is illustrated by way of example.

[0011] Figure 5 A block diagram of an embodiment of a method for determining a position of a camera is illustrated by way of example.

[0012] Figure 6 A block diagram of an interface for performing operations is illustrated by way of example.

[0013] Figure 7 A schematic diagram of an embodiment of a user interface for eye contact assistance configuration is illustrated by way of example.

[0014] Figure 8 A schematic diagram of an embodiment of a method for eye contact assistance in video conferencing is illustrated by way of example.

[0015] Fig. 9 A block diagram of an embodiment of a machine (eg, a computer system) for implementing one or more embodiments is illustrated by way of example. DETAILED DESCRIPTION

[0016] In the following description, reference is made to the accompanying drawings which form a part thereof, and specific embodiments that can be put into practice are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that other embodiments may be utilized without departing from the scope of the embodiments, and structural, logical and / or electrical changes may be made. Therefore, the following description of the embodiments should not be carried out in a limited sense, and the scope of the embodiments is limited by the appended claims.

[0017] Through technology, people are spending more time in conference calls than in face-to-face meetings. Often, the position of a web camera relative to a video conference display makes it appear to the user that they are not participating in the meeting. For example, in some configurations, the focus of the web camera is on one side of the participant's face, forehead, or chin, or not on the participant at all. These configurations may provide the appearance that the participant is not participating, regardless of whether the participant is actually participating.

[0018] The various embodiments are implemented on communication platforms (Microsoft WebEx, WebEx, GoToMeeting, etc.) provide a mode in which the presentation of information during a meeting can be used to help users maintain eye contact. Eye contact improvement can be achieved by presenting information near the location of the camera on the display. By changing the content presented near the camera or presenting interesting information to the attendees, the attendees will naturally tend to look near the camera, thereby providing important eye contact. "Close" or "close" in this context means as close to the camera as possible given the frame and the location of the web camera.

[0019] In one example, the location of a camera (sometimes referred to as a web camera) may be determined. A user (sometimes referred to as a participant) may provide information indicating the location of the camera relative to a display, or the camera location may be inferred. Embodiments may then provide support in a location near the camera, such as surface scripts, notes, potential questions, chat entries, active participant videos, and the like; present notifications about incoming calls, instant messages, incoming emails, upcoming meetings, and the like near the camera location; position a chat window near the camera location; provide “nudges” to the user to look at the camera more frequently (e.g., based on machine learning (ML) inferences about eye contact from camera signals); or provide context-relevant recommendations based on meeting notes, participants, and the like near the camera location.

[0020] Embodiments may be adapted to a user's preferred engagement style based on the active context of the meeting. Context in context includes whether the user is speaking, whether the user is providing content (e.g., sharing a document, screen, chat, etc.), whether the camera is turned on, etc. Embodiments may provide information around the location of the web camera during an online meeting to increase the user's perception of engagement with the camera. Embodiments help create a better perception of eye (camera lens) contact, leading to better meeting outcomes.

[0021] Further details of the embodiments will now be described with reference to the drawings. The drawings illustrate examples of the embodiments, and one or more components of one embodiment may be used with or in place of components of a different embodiment.

[0022] Figure 1A block diagram of an embodiment of a video conferencing system 100 is illustrated by way of example. System 100 is merely an example of a video conferencing system, and many other variations of display and camera configurations are possible. For example, many modern systems include multiple displays whose content is controlled by a single operating system. Other modern systems include a camera 110 built into a bezel 112 of a display 114 (e.g., such as modern laptop computers, teleprompters, other modern systems integrated with a camera located behind or in the main display area 102 or communicatively coupled to other portions of the main display area 102 or display 114 (e.g., such as modern smart phones, tablet computers, etc.), among many other configurations.

[0023] The system 100 includes a camera 110 communicatively coupled to a display 114. The display 114 provides a medium through which a user can interact with a user interface including a primary display area 102 and a secondary display area 104. The size and position of the display areas 102, 104 are configurable. The display areas 102, 104 are controlled by the video conferencing application. Not all video conferencing applications provide multiple display areas 102, 104, so the secondary display area 104 is optional.

[0024] The display 114 provides a view of the main display area 102 and the secondary display area 104. The main display area 102 provides a view of the current speaker, shared documents, the conference organizer, etc. The main display area 102 is generally what the participants are viewing when they are consuming the video conference. The main content of the meeting is provided through the main display area 102, or the person who is speaking controls the content presented in the main display area 102.

[0025] The display 114 also includes a secondary display area 104. The secondary display area 104 may include icons or avatars representing individual attendees of the meeting, camera feeds of attendees, or a combination thereof. As a simple illustrative example, the secondary display area 104 as shown includes icons 108A, 108B of individual attendees. Although the secondary display area 104 is shown on the main display area 102, the secondary display area 104 may be located in various other locations, such as outside the main display area 102.

[0026] The display 114 also includes information 106 presented near a camera 110 built into or coupled to the display 114. Presenting the information 106 near the camera 110 may cause the user to look toward the camera 110. Looking toward the camera 110 causes the perception that the user is participating in a video conference. The information 106 is shown as being separate from the primary display area 102 and the secondary display area 104. However, the information 106 may include the content of the primary display area 102 or the secondary display area 104, such that the primary display area 102 or the secondary display area 104 is presented as the information 106.

[0027] The bezel 112 of the display 114 is the physical border around the display 114. The bezel 112 is the boundary between the screen providing a view of the image and the frame or housing of the electrical / electronic components of the display 114, smartphone, or any other computing device.

[0028] Figure 2 A block diagram of an embodiment of a system 200 for eye contact assistance in video conferencing is illustrated by way of example. The system 200 as shown includes an information generator 234, and one or more data sources including conference environment data 220, information and triggers 228, external data 238, a knowledge graph 256, and a participant profile 240.

[0029] The information generator 234 determines the camera position 226. Determining the camera position 226 may include looking up the camera position in the external data 238. Determining the camera position 226 may include requesting the participant 260 to look at the camera 110, such as through an audio prompt, a visual prompt, a tactile prompt, or a combination thereof, and using gaze tracking to determine where the participant is looking. Gaze tracking will be discussed in more detail elsewhere. Determining the camera position 226 may include generating and analyzing a histogram of the gaze positions of the participant 260 during the video conference. For many participants, the gaze of the participant 260 may be indicated by a spike in the histogram of the gaze positions. Determining the camera position 226 may include using a trained machine learning (ML) model to determine the position of the camera 110. The ML model may be trained using video data that is marked as indicating the position of the camera. The ML model may then receive additional video data and provide a classification of the camera position as an output.

[0030] Receiving camera position 226 may include receiving data from device 224 indicating that camera 110 (see Figure 1 ) with respect to the position of the display 114. The attendee 260 may indicate the camera position 226, such as by clicking on a picture of the display configuration. The camera position 226 may include data indicating the number of displays, the configuration of the displays relative to each other, an identifier indicating the display(s) closest to the camera 110, position information indicating the position on the display(s) closest to the camera 110, and the like. The device 224 may determine the monitor configuration via an operating system of a control panel of the device 224 and provide the monitor configuration to the attendee.

[0031] The information generator 234 can be coupled to the presenter attendee device 224. The information generator 234 can provide a personalized view (PV) 248 to the conference attendee 260. The PV 248 can configure the information presentation on the user interface presented by the device 224. The PV 248 can include data configured by the information generator 234 to help the attendee 260 increase eye contact with the camera 110 during the meeting. The information generator 234 provides data that causes the device to present the information 106. The information 106 is a graphical display of data that captures the attention of the attendee 260, causing the attendee to view the graphical display of the information 106.

[0032] The information generator 234 may be based on the image data from the camera 110 (in Figure 2 110). The information generator 234 may determine whether the participant 260 is making eye contact with the camera 110 using a gaze destination (referred to as an audio / video (A / V) stream 222 in the example embodiment) of the present invention. Determining eye contact may include comparing a gaze destination to a camera position. If the gaze destination is nearby (e.g., within a specified distance or angle of the camera position 226), then it is determined that the user is looking at the camera 110. In response to determining that the participant 260 is not making eye contact with the camera 110, the information generator 234 may cause the information 106 to be displayed near the camera 110. In some embodiments, the information 106 may be displayed near the camera 110 regardless of whether the participant 260 is making eye contact with the camera 110. In such embodiments, the information generator 234 (or other components of the system 200) need not determine whether the participant 260 is making eye contact with the camera 110. The information generator 234

[0033] The meeting environment data 220 can be accessed by the information generator 234. The meeting environment data 220 can include the date, time, meeting agenda, list of invitees, list of required attendees, list of optional attendees, attendee device information (e.g., manufacturer, model, capabilities, etc.), presentation (e.g., files or other computing resources to be presented), etc.

[0034] Information generator 234 may receive A / V stream 222 from device 224. Such an A / V stream is optional because the conferencing application operating device 224 may include video data of participant 260, as this data is typically aggregated at a central server (e.g., in the cloud) and provided to device 224.

[0035] The information generator 234 may determine the emotions of the attendee 260, prompts from the attendee 260, and the like. The prompts may include spoken word phrases in audio, gestures in video, objects in video, and the like. For example, the attendee 260 may respond to prompts, sometimes referred to as triggers, and cause the personalized message to be provided as the information 106. In another example, the attendee 260 may respond to a specified gesture (e.g., "OK", a swipe in a specified direction, pointing to a specified object, clicking a software control with a mouse, touching their head, scratching their face, blinking, or other hand, facial, or body gestures) to cause the personalized message to be provided. In another example, the attendee 260 may respond to a specified object appearing in the stream 222 to cause the personalized message to be provided. The object may be present in an image in a presentation file, an object placed by the presenter in the field of view of their camera, and the like. The personalized message may be provided near the camera 110, such as to help the attendee 260 maintain eye contact with the camera 110.

[0036] The language generator 234 may perform object recognition, speech recognition, gesture recognition, or a combination thereof using a plurality of means (such as, for example, a plurality of different machine learning models) for performing the following items. Object, speech, or gesture recognition may help determine whether the conditions indicated by the defined trigger have been met. The trigger may then cause the information 106 to be presented near the camera 110, or may prohibit the information 106 from being presented near the camera 110. The embodiments are not limited to specific techniques for object, speech, or gesture recognition. Some examples of these techniques are provided only for the context of the meaning of object, speech, and gesture recognition. Example object detection techniques include those using computer-aided drawing (CAD), such as object models, edge comparison-based models, grayscale matching, gradient matching, histograms of receptive field responses, interpretation trees, posture consistency, posture clustering, invariance, geometric hashing, scale-invariant feature transform (SIFT), speeded-up robust features (SURF), word bags, neural networks (NN), etc. Example gesture recognition techniques include techniques based on three-dimensional (3D) models, skeleton-based techniques, appearance-based techniques, electromyography-based techniques, etc. Example speech recognition techniques include Hidden Markov Models, Dynamic Time Warping based speech recognition, NN, and end-to-end automatic speech recognition.

[0037] The attendee 260 or an agent of the attendee 260 may define information prompts that may be stored in the information and trigger database 228. During meeting setup, meeting acceptance, before or during a meeting, a wizard may be presented by the device 224. The wizard may provide the attendee 260 with options for providing information 106. A wizard (sometimes referred to as a setup assistant) is software that provides a user interface that presents a series of dialog boxes to a user to guide the user through a series of well-defined steps. Because the data available to the information generator 234 may be limited, and the form of the information 106 provided in the PV 248 may also be limited, the setup assistant may guide the user through possible information presentations and prevent the user from attempting to use unsupported conditions or types.

[0038] Using a UI, wizard, or setup assistant provided as part of the conference application, attendees 260 can define information criteria. Information criteria can include an indication of the content of information 106, one or more conditions that cause information 106 to be presented to attendees when satisfied, recipients of information 106, and the form of information 106. Conditions can include time of day, passage of time, object detection, gesture detection, phrase detection, gaze deviation from camera 110, combinations thereof, and the like. The condition can be detected by a gaze detector operating in conjunction with a user-configurable timer based on best practices for participating presenters, etc. Information 106 can include images, text, audio, video, clickable hyperlinks or other software controls such as buttons or other data to be provided to the person indicated as the recipient. Information 106 can be associated with criteria (sometimes referred to as characteristic conditions because it is a condition that a user has a specified characteristic feature or qualification) so that if the user meets the criteria, they receive the information 106. Some of the information may be default content (no criteria are required to receive that information in the vicinity of the camera 110), while other information may be associated with criteria that, when met, indicate that the information 106 is to be provided to the attendee 260. The information 106 criteria may be profile parameters derived from or explicitly provided in the user profile of the attendee 260. The profile parameters may indicate job title, education status, expertise, task status, device type, current location about the attendee, and other information available in the attendee profile, etc.

[0039] The form of the information may include an indication of how the information is presented. The form may include the type of information 106, such as a text box, thought bubble, video (e.g., as a thumbnail or on the primary display area 102 or the secondary display area 104, etc.), a graphic, a scrolling text box, etc. The form of the information 106 may include transparency, font, size of the text or video, color, location of the message on the display (e.g., the display, location on the display, such as the background, corner, above the presentation, or a combination thereof), a background pattern or image of the information 106, a customizable indicator provided with the information 106 that indicates to the attendees 260 that they have a message (e.g., a visual, audio, or tactile indicator), a combination thereof, etc. The recipient may be any attendee, attendee, or presenter attendee. The recipient may be defined by name, email address, profile parameters, etc.

[0040] Information generator 234 may receive information and trigger data 230 from information and triggers 228, profile data 242 from attendee profiles 240, other data 236 from external data 238, or a combination thereof. External data 236 is any data related to attendee 260 but accessed from outside the organization where the person setting up the meeting works. External data 236 may include public data, such as may be obtained from social media platforms, county, state, or city records, Census Bureau or other government organizations, electronic mail (email) servers, combinations thereof, and the like. Profile data 242 is data related to attendees and accessed from within the organization. Profile data 242 may include job positions, task completion (e.g., as part of a larger project or group), knowledge graphs, combinations thereof, and the like.

[0041] The information generator 234 may use the external data 236, the profile data 242, the knowledge graph data 258, or a combination thereof to determine whether a condition in the information and trigger data 230 has been satisfied. In response to the condition(s) being satisfied, the information generator 234 may enhance the A / V stream 222 to include the information 106 indicated by the information and trigger data 230.

[0042] When determining whether there is any information 106 associated with the satisfied condition, the information generator 234 can issue a query to the information and trigger 228, which returns whether any item in the external data 236, the profile data 242, or any combination thereof satisfies the condition of the information 106. The information and trigger 228 can return any information associated with the condition that has been satisfied, as well as data indicating the recipient of the information and the format of the information. The information generator 234 can then add the information to the A / V stream of each recipient and provide the PV 248 to the corresponding participant 260.

[0043] The information generator 234 may include electrical or electronic components arranged or programmed to perform its operations. The electrical or electronic components may include one or more transistors, resistors, capacitors, diodes, amplifiers, switches, inductors, multiplexers, logic gates (e.g., AND, OR, NOT, INVERT, buffers, etc.), power supplies, memory devices, processing units (e.g., central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.), analog-to-digital converters (ADCs), digital-to-analog converters (DACs), regulators, oscillators, etc.

[0044] In some embodiments, the system 200 can generate messages and triggers and automatically provide (without human intervention after deployment) corresponding PV 248. Automatic information can be a supplement or alternative to manually defined information presentation. Automatic generation of information can include the message generator 234 analyzing previous messages and triggers 228, external data 238, attendee profiles 240, meeting agendas, knowledge graphs 258, combinations thereof, etc.

[0045] The knowledge graph 256 is not a database in the traditional sense of the word. The knowledge graph 256 is not based on a relational or object database model, but rather a graph database that maps the relationships between users and their various activities. Graph databases are based on entities (or "nodes") and mapping relationships (or "links") between them. The knowledge graph is a good match for Web content because, in a way, the Web itself is a graph database with pages as nodes and relationships represented by hyperlinks connecting them. Knowledge graphs and Satori from Microsoft are semantic data objects, each of which has a unique identifier, a set of attributes based on the attributes of the real-world subject they represent, and links representing relationships from the subject to other entities. Entities also include actions that someone searching for that subject might want to take. Knowledge graph data 258 can be provided by the knowledge graph 256.

[0046] The information generator 234 can interface with one or more applications 262, such as video conferencing applications, web browsers, personal information managers (PIMs), financial applications, sports applications, social media applications, computer-aided drawing applications, presentation generation applications, and many other application types. The application 262 can provide information data 264 that the information generator 234 (e.g., in response to a request or query from the information generator 234, or without a prompt from the information generator 234) can present as information 106 on the device 224.

[0047] Information data 264 or information 106 may include contextually relevant information. Contextually relevant information may take the form of enhanced or supplemental messages related to (i) a seminar (sometimes referred to as a "meeting"), (ii) a current conversation to help reach a point (e.g., a message "For more information, see here." sent to all or only a subset of the participants), (iii) the current point in time (e.g., a "It's time to move on" message sent to the speaker only when there are many items on the agenda), or (iv) a message that may be left by the speaker to remind themselves of what they should do during the meeting (e.g., "Ask everyone on the team for their perspective"). These contextual messages may support the speaker (e.g., helping them reach a point or providing supplemental information, such as reminders, hyperlinks, or bulleted lists) as well as others on the call (e.g., getting additional information and reminders).

[0048] There are at least three modalities for contextual messaging, such as (1) messages shared by the speaker for use by others on the call, (2) messages shared directly to the current speaker for use only by the current speaker, and (3) messages automatically generated by the conferencing technology (e.g., based on user-provided rules).

[0049] Messages in any modality can be guided by one or more of the following: (a) Organizational strategy and supporting data. For example, all people who have not completed their training could see the following message during an HR call: "Reminder: Complete your required training by <date>". (b) User specification. For example, the content of the message can be provided by the speaker or meeting manager before the meeting, and even tagged with specific key phrases and / or timing to trigger the message. (c) System inference. For example, the content of the message can be automatically derived from the meeting agenda, meeting minutes, or communications prior to the meeting (such as reminders of commitments or requests made in emails).

[0050] The format of messages (how they are presented) can vary depending on the nature of the message, the nature of the call (e.g., formal meeting vs. informal chat), and the environment. The format can be user-defined, or vary based on inferences about the best way to convey the message in the current context. Messages can also be actionable (e.g., via a clickable hyperlink).

[0051] Participant 260 may provide configuration data 266 to information generator 234, such as through device 224. Configuration data 266 may indicate a trigger, whether eye contact assistance is on or off, information data 264 is to be presented as information, context, etc. More details about configuration data Figure 7 etc. provided.

[0052] Figure 3 By way of example, a perspective view of an embodiment of a participant 260 is shown, the participant 260 making insufficient eye contact. Figure 4 , the gaze of participant 260 (indicated by arrow 330) is on object 332 that is not near camera 110. Information generator 234 or another component of computing device 224 can determine the object that participant 260 is looking at. There are many ways to determine gaze, including using gaze models, gaze features, object recognition, image filtering (e.g., Gabor wavelets), color distribution, head pose, or a combination thereof, identifying an eye, determining the center of the iris or pupil of the identified eye, and projecting along a line of sight (sometimes called a visual axis) through the center of the pupil.

[0053] exist Figure 3 In the example of FIG. 3 , participant 260 may be perceived as being uninterested in the content being presented on user interface 336 of the video conference. The perception of uninterest may affect how other participants perceive participant 260 in terms of professionalism, respect, politeness, etc. The perception of uninterest may affect whether participant 260 gets a job, gets a promotion, is invited to an event, etc.

[0054] To help reduce perceptions of disinterest, computing device 224 can help keep attendee 260's gaze near camera 110. The time between gazes can be user configurable, based on best practices, etc. The time between eye contact can be reduced by presenting attention-capturing information 106 near camera 110. Quantification of gaze time can be provided retrospectively, such as after a video conference. Quantification can provide part of a report on meeting engagement (you only looked at the camera 20% of the time while speaking).

[0055] One or more components of device 224 (such as processing circuitry (see Fig. 9 ) and information generator 234) can determine whether the gaze of the participant is near camera 110. In this context, "near" means within a specified distance, within a specified number of pixels, within a defined pixel neighborhood, etc. In some cases, the angle 338 between the line of sight of participant 260 (indicated by line 330) and the line of sight of camera 110 (indicated by line 334) can be used to determine whether the gaze is near camera 110. The user can specify the maximum angle considered near camera 110, or a default specified angle can be used to determine whether the gaze is near camera 110.

[0056] In response to determining that participant 260 is not looking near camera 110, object 332 (object 332 is an example of information 106) that the user is looking at can be moved on user interface 336 so that object 332 is near camera 110. Then, while participant 260's gaze remains on object 332, the gaze ends near camera 110. The gaze near camera 110 provides a perception that participant 260 is engaged with the presentation.

[0057] Figure 4 A block diagram of an embodiment of the system is illustrated by way of example after an object 332 has been moved to be closer to the camera 110. By presenting the object 332 near the camera 110, an angle 338 between the line of sight of the participant 260 and the line of sight of the camera 110 is reduced. Whether the object 332 is close to the camera 110 can be determined by comparing the angle 338 to a standard or presenting the object 332 as close to the camera 110 as possible. As close to the camera 110 as possible means presenting the object 332 at a pixel of the user interface 336 closest to the camera 110. The standard can include a default or user-specified angle, such as 15 degrees, 20 degrees, 30 degrees, a smaller number of degrees, or a number of degrees within this interval.

[0058] In some embodiments, the user interface 336 can be presented using foveated rendering. Foveated rendering forces a portion of the screen to be in focus and the rest of the screen to be out of focus. Using foveated rendering, the gaze of the participant 260 tends to be on the portion of the user interface 336 that is in focus.

[0059] Figure 5 A block diagram of an embodiment of a method 500 for determining the location of a camera 110 is illustrated by way of example. At operation 552, the information generator 234 or another component of the device 224 may determine whether the device 224 has a built-in camera. Operation 552 may be performed based on the manufacturer and model 550 of the device 224. The manufacturer is the manufacturer of the device 224, such as HewlettPackard, Apple, Dell, Microsoft, Lenovo, etc. The model indicates a specific configuration of the components of the device 224. For example, iPhone X, Latitude 9520, etc.

[0060] The information generator 234 performs operation 552, such as by querying the external data 238 to determine whether the device 224 includes a built-in camera. If it is determined at operation 552 that the device 224 does have a built-in camera, the information generator 234 can use the location of the built-in camera as the location of the camera at operation 554. If it is determined at operation 552 that the device 224 does not have a built-in camera, the information generator 234 can ask the user to specify the location of the camera 110 at operation 556.

[0061] Note that operation 552 is only one way to infer where the camera 110 is located. Other ways to infer where the camera is located include asking the participant 260 to look at the camera 110, determining the location of the gaze, verifying based on video data from the camera 110 that the gaze includes a head pose corresponding to facing the camera 110, generating a histogram of the participant's 260 gaze, combinations thereof, and the like.

[0062] Figure 6 A block diagram of an interface 600 for performing operation 556 is illustrated by way of example. Interface 600 provides a depiction of a display configuration of participant 260. The display configuration may be determined by operating system setting data. Figure 6 In the example of , participant 260 has a rather complex display setup, which includes four displays numbered "1", "2", "3", and "4". The user may be prompted to select the position of the camera relative to his current display configuration. Figure 6 In the example of , the conferees 260 are prompted to move their mouse pointer 660 to the location of the camera 110 relative to the display and right-click to indicate the approximate location of the camera 110. Figure 6 In the example of , participant 260 has selected a position centered between displays 3 and 1 .

[0063] exist Figure 6 In the example of , information generator 234 may (e.g., in response to determining that a video conference is occurring) present information 106 in area 662 near camera 110. This will cause participant 260 to look near camera 110 and at least provide the appearance that the participant is participating in the video conference by increasing the appearance of eye contact with the camera.

[0064] During a video conference, many participants have different things they prefer to focus on. One participant may prefer to focus on the active speaker, another participant may prefer to focus on chatting, another participant may prefer to focus on the content being presented, and yet another participant may prefer to focus on emails or other items received from an external application received during the video conference. In some embodiments, participants 260 can specify their preferences for content presented near the location of camera 110.

[0065] Figure 7 A diagram of an embodiment of a user interface 700 for an eye contact assistance configuration is shown by way of example. Although the eye contact assistance interface 700 is shown and described as including radio buttons and a text input box, the interface 700 can be implemented using one or more drop-down menus, sticky menus, scrolling panels, card-style selection interfaces, forms, steppers, switches, sliders, buttons, autofillers, combinations thereof, and the like.

[0066] Using interface 700, conferee 260 may turn on eye contact assistance by selecting (e.g., physically, such as by a mouse, touch screen, touchpad, pen, stylus, etc., such as by a voice command, tactile selection such as by a Braille interface, etc.) software control 772. The conferee may turn off eye contact assistance by selecting software control 774. The software controls of interface 700 are shown as radio buttons, but may be different software controls that allow a user to configure eye contact assistance performed by information generator 234 or another component of device 224.

[0067] Conferee 260 may select or otherwise indicate what information to present near camera 110 by using software controls 780, 782, 784, or a combination thereof. When selected, software control 780 causes items marked as high importance, urgent, having a short deadline, etc. to be presented near camera 110. Software control 782, when selected, causes the active speaker of the video conference, content, live chat, etc. to be presented near camera 110. Conferee 260 may use software control 784 to define different content for presentation near camera 110. Other content may include graphics, text, content from applications other than the video conferencing application that is not necessarily urgent, combinations thereof, and the like.

[0068] Using software controls 786, 788, 790, 792, 794, the conferee 260 can define a trigger (or lack of trigger) that causes the information 106 indicated by the software controls 780, 782, 784 to be presented near the camera 110. Software control 786, when selected, allows the conferee 260 to define an utterance that, when detected, causes the information generator 234 to present the information 106 near the camera 110. Software control 788, when selected, allows the conferee 260 to define a physical input that, when detected, causes the information generator 234 to present the information 106 near the camera 110. Software control 790, when selected, causes the information generator 234 to present the information 106 near the camera 110 in response to detecting that the gaze of the conferee 260 is straying away from the camera 110 (more details on gaze straying are provided in the accompanying drawings). Figures 3 to 4 is provided).

[0069] The software control 792, when selected, causes the information generator 234 to provide the information 106 in the vicinity of the camera 110, regardless of whether a trigger is detected. In an embodiment that includes the software control 792, the information generator 234 can be set to a default setting that provides a view of the video conference regardless of the position of the camera 110. The software control 794, when selected, allows the conferee 260 to define a trigger that cannot be defined using the software controls 786, 788, 790, 792. Another trigger may include detection of a gesture in the A / V stream 222, detection of an object in the A / V stream 222, a composite trigger that is a combination of triggers, and the like.

[0070] The participant 260 can specify whether the camera position 226 is to be inferred (using software controls 796) or specified by the participant 260 (using software controls 798). Inferring the camera position 226 can include: asking the participant 260 to look at the camera 110, determining one or more pixels on the display closest to the gaze, and using the determined one or more pixels as the camera position 226; using a trained machine learning model to infer the position of the camera 110; analyzing video data from the camera 110 to determine the camera position relative to the user (e.g., such as by using world rays, etc.); generating a histogram of the user's gaze and determining the camera position 226 based on the user's gaze; and the like. The user can specify the camera position 226 in a variety of ways, including: selecting a position for the camera 110, such as with respect to the user's gaze; Figure 6 described; a user provides a natural text or audio description of the camera position 226; and the like.

[0071] Figure 8 A diagram of an embodiment of a method 800 for eye contact assistance in video conferencing is illustrated by way of example. As shown, the method 800 includes: determining a position of a web camera relative to a display at operation 880; providing a user interface for video conferencing on the display at operation 882; and providing a graphical display of information at a pixel of the display closest to the position on the user interface at operation 884.

[0072] The method 800 may also include receiving video data from a web camera. The method 800 may also include determining a gaze direction of a user in the video data based on the video data. The method 800 may also include inferring a position of the web camera based on the gaze direction.

[0073] Operation 880 may include prompting a user to provide a location by a user interface; and receiving the location from the user. Operation 884 may include receiving trigger data. Operation 884 may include comparing the trigger data to determine that a condition is satisfied. Operation 884 may include providing information by a pixel in response to determining that a condition is satisfied.

[0074] The method 800 may also include receiving, by the user interface, information data indicating content of the information from the user. The method 800 may also include receiving, by the user interface, form data indicating how to present the content from the user. The method 800 may also include presenting, by the user interface, the content in the form indicated by the form data. The information may include video data of an active speaker, content of a chat window or a shared screen. The information may include data received by a second application that is different from the application providing the video conferencing functionality.

[0075] Method 800 may also include determining that the user is not looking at the camera. Method 800 may also include providing information at the pixel in response to determining that the user is not looking at the camera. Determining that the user is not looking at the camera may include determining that the user has not looked at the camera for a specified amount of time.

[0076] Fig. 9 A block diagram of an embodiment of a machine 900 (e.g., a computer system) for implementing one or more embodiments is shown by way of example. System 100, camera 110, display 114, device 224, information generator 234, application 262, etc. may include one or more components of machine 900. One or more of system 100, camera 110, display 114, device 224, information generator 234, application 262, method 500, interface 336, interface 600, interface 700, method 800, or components or operations thereof may be implemented, at least in part, using components of machine 900. An example machine 900 (in the form of a computer) may include a processing unit 902, a memory 903, a removable storage device 910, and a non-removable storage device 912. Although an example computing device is shown and described as machine 900, in different embodiments, the computing device may be in different forms. For example, the computing device may alternatively be a smartphone, a tablet, a smartwatch, or include a computer system that is compatible with the computing device. Fig. 9 Other computing devices that may include the same or similar elements shown and described herein. Devices such as smart phones, tablet computers, and smart watches are generally referred to as mobile devices. In addition, although various data storage elements are shown as part of machine 900, the storage device may also or instead include a cloud-based storage device accessible via a network (such as the Internet).

[0077] The memory 903 may include volatile memory 914 and non-volatile memory 908. The machine 900 may include - or have access to - a computing environment that includes - various computer-readable media, such as volatile memory 914 and non-volatile memory 908, removable storage 910, and non-removable storage 912. Computer storage includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) & electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0078] The machine 900 may include or have access to a computing environment including an input 906, an output 904, and a communication connection 916. The output 904 may include a display device that may also be used as an input component, such as a touch screen. The input 906 may include a touch screen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated into the machine 900 or coupled to the machine 900 via a wired or wireless data connection, and one or more of other input components. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as a database server, including cloud-based servers and storage devices. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), Bluetooth or other network.

[0079] Computer-readable instructions stored on a computer-readable storage device may be executed by a processing unit 902 (sometimes referred to as a processing circuit) of the machine 900. Hard drives, CD-ROMs, and RAM are some examples of items that include non-transitory computer-readable media such as storage devices. For example, the computer program 918 may be used to cause the processing unit 902 to perform one or more methods or algorithms described herein.

[0080] In some embodiments, the operations, functions or algorithms described herein may be implemented in software. Software may include computer executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transitory memories (e.g., non-transitory machine-readable media) or other types of local or networked hardware-based storage devices. In addition, such functions may correspond to subsystems, which may be software, hardware, firmware, or a combination thereof. Multiple functions may be performed in one or more subsystems as needed, and the described embodiments are merely examples. Software may be executed on a processing circuit, such as may include a digital signal processor, an ASIC, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or other types of processors operating on a computer system (such as a personal computer, a server, or other computer system), thereby transforming such a computer system into a specially programmed machine. Additionally or alternatively, a processing circuit may include electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, field programmable gate arrays (FPGAs), etc.). The terms computer-readable medium, machine-readable medium, and storage device do not include carrier waves or signals, which are considered to be too transitory.

[0081] Additional Notes and Examples

[0082] Example 1 includes a system comprising: a processing circuit; and a memory, the memory comprising instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for assisting eye contact with a web camera in a video conference, the operations comprising: determining a position of the web camera relative to a display; providing a user interface for the video conference on the display; and providing a graphical display of information on the user interface at a pixel of the display that is closest to the position.

[0083] In Example 2, Example 1 also includes, wherein the operations further include receiving video data from a web camera, determining a gaze direction of a user in the video data based on the video data, and inferring a position of the web camera based on the gaze direction.

[0084] In Example 3, at least one of Examples 1-2 also includes, wherein determining the location of the web camera comprises prompting a user, by a user interface, to provide the location; and receiving the location from the user.

[0085] In Example 4, at least one of Examples 1 to 3 also includes wherein providing the graphical display of the information includes receiving trigger data, comparing the trigger data to determine that a condition is satisfied, and in response to determining that the condition is satisfied, providing the information by the pixel.

[0086] In Example 5, at least one of Examples 1 to 4 further includes, wherein the operation further includes receiving, by the user interface, information data from a user indicating content of the information.

[0087] In Example 6, Example 5 also includes, wherein the operations further include receiving, by the user interface, form data indicating how to present the content from a user, and presenting, by the user interface, the content in a form indicated by the form data.

[0088] In Example 7, at least one of Examples 5 to 6 further includes, wherein the information includes video data of an active speaker, content of a chat window, or a shared screen.

[0089] In Example 8, at least one of Examples 5 to 7 further includes, wherein the information includes data received by a second application, the second application being different from the application providing the functionality of the video conference.

[0090] In Example 9, at least one of Examples 1 to 8 further includes, wherein the operations further include determining that the user is not looking at the camera, and in response to determining that the user is not looking at the camera, providing the information at the pixel.

[0091] In Example 10, Example 9 further includes, wherein determining that the user is not looking at the camera includes determining that the user has not looked at the camera for a specified amount of time.

[0092] Example 11 includes a method for performing the operations of the processing circuit of at least one of Examples 1-10.

[0093] Example 12 includes a machine-readable medium including instructions that, when executed by a machine, cause the machine to perform the operations of the method of Example 11.

[0094] Although several embodiments have been described in detail above, other modifications are possible. For example, the logical flows depicted in the accompanying drawings do not require the order or sequential order shown to achieve the desired results. Other steps can be provided, or steps can be eliminated from the described flows, and other components can be added to or removed from the described systems. Other embodiments may be within the scope of the appended claims.

Claims

1. A system comprising: Processing circuit; a memory comprising instructions that, when executed by the processing circuit, cause the processing circuit to perform operations for eye contact assistance with a web camera in a video conference, the operations comprising: determining a position of the web camera relative to a display; providing a user interface for the video conference on the display; and A graphical display of information is provided on the user interface at the pixel of the display that is closest to the location.

2. The system of claim 1, wherein the operations further comprise: receiving video data from the web camera; determining a gaze direction of a user in the video data based on the video data; as well as The position of the web camera is inferred based on the gaze direction.

3. The system of claim 1 , wherein determining the location of the web camera comprises: Prompting the user to provide the location by the user interface; and receiving the location from the user.

4. The system of claim 1 , wherein the graphical display providing information comprises: Receive trigger data; comparing the trigger data to determine that a condition is met; as well as In response to determining that the condition is satisfied, the information is provided by the pixel.

5. The system of claim 1, wherein the operations further comprise: Information data indicative of content of the information is received by the user interface from a user.

6. The system of claim 5, wherein the operations further comprise: receiving, by the user interface, from the user, format data indicating how the content is to be presented; as well as The content is presented by the user interface in a form indicated by the form data.

7. The system of claim 5, wherein the information includes video data of an active speaker, a chat window, or content of a shared screen.

8. The system of claim 5, wherein the information includes data received by a second application, the second application being different from the application providing functionality of the video conference.

9. The system of claim 1, wherein the operations further comprise: determining that the user is not looking at the camera; as well as In response to determining that the user is not looking at the camera, providing the information at the pixel.

10. The system of claim 9, wherein determining that the user is not looking at the camera comprises determining that the user has not looked at the camera for a specified amount of time.

11. A method for eye contact assistance in video conferencing, the method comprising: determining a position of the web camera relative to a display; providing a user interface for the video conference on the display; as well as A graphical display of information is provided on the user interface at the pixel of the display that is closest to the location.

12. The method according to claim 11, further comprising: receiving video data from the web camera; determining a gaze direction of a user in the video data based on the video data; as well as The position of the web camera is inferred based on the gaze direction.

13. The method of claim 11, wherein determining the location of the web camera comprises: Prompting the user to provide the location by the user interface; and receiving the location from the user.

14. The method of claim 11, wherein providing the graphical display of information comprises: Receive trigger data; comparing the trigger data to determine that a condition is met; as well as In response to determining that the condition is satisfied, the information is provided by the pixel.

15. A machine-readable medium comprising instructions which, when executed by a machine, cause the machine to perform the operations of the processing circuit according to one of claims 1 to 10.

Images