Virtual eye contact in video interactions

Abstract

A first video of at least an eye of a first person looking along a video capture axis is captured. A second video of at least an eye of a second person is presented along a display axis. The video capture axis is aligned along a common axis with the display axis such that the first person can look along the common axis at the second video of at least the eye of the second person when the first video of the eye of the first person is captured along the common axis. Instructions are executed to maintain an eye contact video interaction session between the first person and the second person using the captured video and the presented video.

Description

Technical Field

[0001] This document covers virtual eye contact in video interactions. Background Technology

[0002] Many traditional in-person, face-to-face interactions (where two or more people are physically close enough to see and hear each other) are being replaced by video interactions. In typical video interactions, two or more physically separated individuals can see and hear each other in real-time digital video and audio presentations. This is achieved, for example, through electronic digital presentations between participants via communication networks such as the Internet.

[0003] Popular scenarios for such video interaction include meetings, video calls, conferences, webinars, medical appointments, parties, social gatherings, courses, podcasts, lectures, seminars, board meetings, discussion groups, and more. The advantages of video interaction include cost savings, reduced travel, less formality, greater spontaneity, flexibility in the number of participants, locations, and categories, and the availability of supplementary features and controls. In 2020, the encouragement of social distancing to reduce the spread of the COVID-19 virus spurred a significant increase in video interaction relative to face-to-face interaction.

[0004] Types of video interaction include video conferencing facilitated by a variety of web-based and software platforms, including FaceTime and other mobile-based systems, as well as video conferencing systems such as Zoom, Skype, Google Meet, and Microsoft Teams.

[0005] In face-to-face interactions, participants can typically look each other in the eye, judge each other's facial expressions and body language, and hear subtle differences in the timbre and clarity of each other's speech.

[0006] In face-to-face interaction, when one person is looking into another's eyes, it can be said that the first person is "projecting eye contact" into the other. Then, if the other person is looking into the first person's eyes, it can be said that the other person is "perceiving" the first person's eye contact. If the other person is not looking into the first person's eyes, then the other person cannot perceive the eye contact projected by the first person. Therefore, when two people are looking at each other during face-to-face interaction, they are each projecting and perceiving eye contact; sometimes it is said that they are engaging in "face-to-face eye contact." In other words, in face-to-face interaction, when one person is perceiving eye contact, eye-to-eye contact between the two people necessarily occurs.

[0007] Eye-to-eye contact is a particularly powerful and engaging feature of live, face-to-face interaction.

[0008] In face-to-face interaction, people project eye contact along the so-called "gaze axis" and perceive eye contact along the so-called "viewing axis." During eye-to-eye contact in the room, the gaze axes and viewing axes of both people are aligned. Summary of the Invention

[0009] In general, in one aspect, a first video of at least the eyes of a first person viewing along the video capture axis is captured. A second video of at least the eyes of a second person is presented along the display axis. The video capture axis and the display axis are aligned along a common axis such that while the first video of the first person's eyes is being captured along the common axis, the first person is able to view the second video of at least the eyes of the second person along the common axis. Instructions are executed to maintain an eye-contact video interaction session between the first and second persons using the captured and presented videos.

[0010] The implementation may include one or a combination of two or more of the following features: The eye-contact video interaction session includes a video conference. A user interface is presented so that the first person can control the eye-contact video interaction session through the user interface. The first video is captured by a discrete camera dedicated to capturing video along the video capture axis. The first video is captured by a camera of a mobile device aligned with the video capture axis. The second video is presented by a discrete display dedicated to presenting video along the display axis. The second video is presented by a display of a mobile device aligned with the display axis. The instructions are executed on a discrete computing device dedicated to executing instructions to maintain the eye-contact video interaction session. The instructions are executed on a computing device that also executes other instructions unrelated to the instructions for maintaining the eye-contact video interaction session.

[0011] In general, in one aspect, the device includes a housing. The housing has: a camera or camera support for a camera or a mobile device having a camera; a video display or display support for a video display or a mobile device having a video display; and an axis alignment device configured to align the video capture axis of the camera with the display axis of the video display along a common axis, such that when the camera is capturing at least the eyes of a first person looking toward the camera along the common axis, the first person can view real-time video of at least the eyes of a second person along the common axis. A computing device or connector is provided on the housing or in the housing's support, the connector for connecting to the computing device or to a mobile device serving as a computing device, the computing device being configured to maintain an eye-contact video interaction session between the first and second persons using the output of the camera and the input of the video display.

[0012] The implementation may include one or a combination of two or more of the following features: The axis alignment device includes a beam splitter. The computing device includes a mobile device. The computing device includes a small computer. The support member is used to hold the housing in a position and orientation such that the common axis extends to a position between the eyes of the first person. The eye contact video interaction session includes a video conference. The eye contact video interaction session is conducted between a first person and a second person in the same building or space. The eye contact video interaction session is conducted between a first person and a second person not in the same building or space. A second computing device is used to execute instructions to provide functions other than maintaining the eye contact video interaction session. The video display includes the video display of the mobile device, which functions as the computing device for maintaining the eye contact video interaction session, and the camera includes a discrete dedicated camera configured to provide video signals to the computing device of the mobile device. The mobile device functions as the computing device for maintaining the eye contact video interaction session, and the mobile device mirrors its display to the video display of the device. The mobile device's camera serves as the device's camera and also as the computing device for maintaining the eye-contact video interaction session. A wireless mouse and keyboard control the computing device. A wireless speaker or microphone, or both, serves as the mobile device's speaker or microphone. Electronic components, including one or more of a microphone, speaker, and computing device, are present in the housing's support structure and base. A light source is present on the housing.

[0013] In general, in one aspect, the device includes: a beam splitter having a front surface and a rear surface. A video sensor and an optical lens are mounted at a position facing the rear surface of the beam splitter. The optical lens has an optical axis extending from the video sensor through the beam splitter. A video display is located at a position facing the front surface of the beam splitter. Mechanisms associated with the beam splitter and the video display implement an open configuration and a closed configuration, in which reflections from the screen of the video display are visible from the position facing the front surface of the beam splitter, and in the closed configuration, reflections from the video display are not visible from the position facing the front surface of the beam splitter.

[0014] The implementation may include one or a combination of two or more of the following features. The mechanism includes a hinge coupled between the beam splitter and the video display and configured to allow relative movement between the beam splitter and the video display in the open and closed configurations. A lamp provides illumination near the device. The lamp is exposed in both the open and closed configurations. A support structure holds the beam splitter, sensor, optical lens, display, and mechanism at a height above the working surface. A housing holds the beam splitter, video sensor, optical lens, video display, and mechanism. A pedestal and base are present, and the housing is supported on the pedestal.

[0015] In general, in one aspect, a handheld device includes: a handle; an assembly connected to the handle and including a camera or camera support for a camera or a mobile device having a camera; a video display or display support for a video display or a mobile device having a video display; an axis alignment device configured to align the video capture axis of the camera with the display axis of the video display along a common axis, such that when the camera is capturing at least the eyes of a first person looking at the camera along the common axis, the first person holding the handle can view real-time video of at least the eyes of a second person along the common axis; and a computing device or connector for connection to the computing device or a mobile device used as a computing device, the computing device being configured to maintain an eye-contact video interaction session between the first person and the second person using the output of the camera and the input of the video display.

[0016] In general, in one aspect, at the first point, the device performs at least the following activities: capturing a first video of at least the eyes of a first person at the first point who is viewing along the video capture axis; presenting a second video of at least the eyes of a second person at the second point along the display axis; aligning the video capture axis and the display axis along a common axis such that while the first video of the first person's eyes is being captured along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis; and executing instructions to maintain a gaze-contact video interaction session between the first person and the second person using the captured video and the presented video. At the second point, the second person controls at least one of the following by communicating information from the second point to the first point: the capture of the first video, the presentation of the second video, and the execution of instructions.

[0017] In general, in one aspect, at the first position, the first device performs at least the following activities: capturing a first video of at least the eyes of a first person at the first position who is viewing along the first video capture axis of the first device; presenting a second video of at least the eyes of a second person at the second position along the display axis of the first device; aligning the video capture axis and the display axis of the first device along a common axis such that when capturing the first video of the first person's eyes along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis; and executing instructions to use the captured video and the presented video and to maintain the first person and the second person at the first position. A video interaction session involving eye contact between the first and third persons; a second device that performs at least the following activities: capturing a third video of at least the eyes of a first person at a first position and viewing along the video capture axis of the second device; presenting a fourth video of at least the eyes of a third person at a third position along the display axis of the second device; aligning the video capture axis and the display axis of the second device along a common axis such that when the third video of the first person's eyes is captured along the common axis, the first person can view the fourth video of at least the eyes of the third person along the common axis; and executing instructions to maintain the video interaction session involving eye contact between the first and third persons using the captured video and the presented video.

[0018] In general, in one aspect, at a first location, the eye-to-eye interaction device performs at least the following activities: capturing a first video of at least the eyes of a first person located at the first location and viewing along the video capture axis; presenting a second video of at least the eyes of a second person located at a second location along the display axis; and aligning the video capture axis and the display axis along a common axis such that when the first video of the first person's eyes is captured along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis. A computing device located at the first location and communicating with the eye-to-eye interaction device executes instructions to maintain a video interaction session of eye contact between the first person and the second person using the captured video and the presented video.

[0019] In general, in one aspect, at least the following activities are performed at a first location: capturing a first video of at least the eyes of a first person at the first location who is viewing along the video capture axis; presenting a second video of at least the eyes of a second person at a second location along the display axis; and aligning the video capture axis and the display axis along a common axis such that when the first video of the first person's eyes is captured along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis. The first video is captured by a camera of a mobile phone or tablet or both, or the second video is presented by a display of a mobile phone or tablet or both. Instructions are executed to maintain a video interaction session of eye contact between the first person and the second person using the captured video and the presented video.

[0020] In general, in one aspect, at least the following activities are performed at a first location: capturing a first video of at least the eyes of a first person at the first location who is viewing along the video capture axis, the first video being captured by a camera of a mobile phone or tablet and communicated by the mobile phone or tablet to a computing device; presenting a second video of at least the eyes of a second person at a second location along the display axis; and aligning the video capture axis and the display axis along a common axis such that when the first video of the first person's eyes is captured along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis. At the computing device, instructions are executed to maintain a gaze-contact video interaction session between the first person and the second person using the captured video and the presented video.

[0021] In general, in one aspect, at a point where a first person and a second person are physically close to each other but unable to engage in on-site eye-to-eye interaction, at least the following activities are performed: capturing a first video of at least the eyes of the first person who is viewing along the video capture axis; presenting a second video of at least the eyes of the second person along the display axis; aligning the video capture axis and the display axis along a common axis such that when the first video of the first person's eyes is captured along the common axis, the first person can view the second video of at least the eyes of the second person along the common axis; and executing instructions to maintain an eye-contact video interaction session between the first person and the second person using the captured video and the presented video.

[0022] In general, in one aspect, at the first location of the healthcare provider, the device performs at least the following activities: capturing a first video of at least the eyes of the healthcare provider who is viewing along the video capture axis; presenting a second video of at least the eyes of a person who is the recipient of healthcare services and is located at a second location along the display axis; aligning the video capture axis and the display axis along a common axis such that when the first video of the provider's eyes is captured along the common axis, the provider is able to view the second video of at least the recipient's eyes along the common axis; and executing instructions to maintain an eye-contact video interaction session between the provider and the recipient using the captured video and the presented video.

[0023] In general, in one aspect, at the first location of the security agent, the device performs at least the following activities: capturing a first video of at least the eyes of a security person viewing along the video capture axis; presenting a second video of at least the eyes of a person seeking access to the first location and located at a second location near the first location along the display axis; aligning the video capture axis and the display axis along a common axis such that when the first video of the security agent's eyes is captured along the common axis, the security agent is able to view the second video of at least the eyes of the person seeking access along the common axis; and executing instructions to maintain a video interaction session of eye contact between the security agent and the person seeking access using the captured video and the presented video.

[0024] Among other advantages, the implementation of eye-contact interaction technology can significantly enhance the video interaction experience between two or more participants by improving the fidelity of all visual and audio components and restoring direct, eye-to-eye horizontal communication between participants. In business contexts, this technology can help improve bidirectional communication between remote participants (e.g., in a work-from-home environment) and imbue such communication with a certain spontaneity and dynamism characteristic of modern open-plan office layouts. Eye-contact interaction devices can operate as standalone devices (requiring no additional hardware) or can be connected to other computers via cable or wirelessly.

[0025] These and other aspects, features, implementations and advantages (a) may be expressed as methods, apparatus, systems, components, program products, business methods, means or steps, and other means for performing the functions, and (b) will become apparent from the following description and from the claims. Attached Figure Description

[0026] Figure 1 , Figure 2 , Figures 5 to 10 , Figure 14 and Figure 15 This is a schematic diagram of an interactive device.

[0027] Figure 3 and Figure 4 It is a three-dimensional view of the workpiece and the assembly housing of the interactive device.

[0028] Figures 11 to 13 This is a schematic diagram.

[0029] Figures 16 to 21 It is the view of the interactive device.

[0030] Figure 22 It is the view of the interactive device. Detailed Implementation

[0031] During video interaction, the projection and perception of eye contact, as well as the alignment of the gaze axis and viewing axis, occur in a different way than in face-to-face interaction. Eye contact that occurs during video interaction is sometimes referred to as "virtual eye contact."

[0032] In video interaction, the display axis of the video monitor and the video capture axis of the camera are parallel but separated by several inches or even feet. Therefore, when a local participant is viewing the video monitor (i.e., aligning their viewing axis with the display axis), their gaze axis is also necessarily aligned with the display axis, not the camera's video capture axis. Consequently, the local participant will not project eye contact to the remote participant, and the remote participant will not perceive eye contact. Similarly, when the local participant aligns their gaze axis with the camera's video capture axis, their viewing axis is also aligned with the video capture axis. Then, the local participant can project eye contact but cannot perceive it, and the remote participant can perceive eye contact but cannot project it back to the local participant. In other words, virtual eye-to-eye contact cannot occur.

[0033] Unlike face-to-face interaction, in video interaction, even if eye-to-eye contact is not possible, eye contact can be perceived or projected. While not as robust and effective as eye-to-eye contact, perceived and projected eye contact that occurs as part of video interaction is useful to participants. Furthermore, in video interaction, the resolution of the video presentation and the fidelity of the sound are limited, especially due to the technological limitations of cameras, displays, microphones, and speakers embedded in computing devices and mobile phones. Supplementary devices such as external cameras, displays, microphones, and speakers can enhance the participant experience in video interaction.

[0034] Unlike face-to-face interactions where participants naturally project eye contact, in video interactions, people must consciously project eye contact by looking directly at the camera. For non-professional participants in video interactions, looking directly at the camera can be strenuous, and for most individuals, their eyes wander or their gaze axis is not aligned with the camera's video capture axis. Even a slight misalignment of the local participant's gaze axis relative to the camera's video capture axis can make remote participants uncomfortable, and thus often eliminates or diminishes a key element of effective human communication in video interactions.

[0035] Overall, for these and other reasons, typical video interactions tend to be less engaging or impactful than in-person, face-to-face interactions, and are often more unsettling and unsatisfying.

[0036] Various technologies have been used to encourage eye contact in specific situations.

[0037] In speech-to-text systems used in the film and television industry, a beam splitter presents text material along the display axis for the user to read, while the participant guides their gaze along the camera's video capture axis. Speech-to-text systems typically present text material on a monitor; the displayed text material is reflected by a partial mirror of the beam splitter, while a camera captures video of the individual reading the material. Thus, the reader appears to be projecting their gaze onto the individual watching the video captured by the camera, which is part of a live broadcast or recorded film or video. Speech-to-text systems do not present the reader with a live image or video of the individual remotely watching the reader read the text material.

[0038] In some inexpensive speech prompters, the integrated display is replaced by the existing display of a smartphone or tablet, and no separate computer is needed to generate the text for display. While smartphones or tablets can provide displays, their embedded cameras cannot simultaneously replace the camera in a speech prompter because the video capture axis is offset relative to the display axis of the monitor, preventing the reader from making eye contact.

[0039] Systems for video interaction using beam splitter technology have been proposed, in which the beam splitter aligns the viewing axis of the participant's view on the display with the gaze axis of the participant captured by the camera.

[0040] In a commercially available device ( https: / / www.bodelin.com / proprompter / product / new- proprompter-desktop / This device uses two mirrors to reflect portions of the display of a computer or mobile device that is presenting a traditional video interaction. The two mirrors are oriented like the mirrors of a periscope so that the participant can see a camera connected to the computer device, thus effectively projecting eye contact onto the person interacting with while also seeing others. The device may be perceived as large and bulky, and obstructing much of the device's display, making it impossible for the participant to see.

[0041] In other proposed devices, one or more cameras are embedded in or associated with a semi-transparent display to enable eye-to-eye contact.

[0042] technology

[0043] like Figure 1 As shown, an eye-contact video interaction technology 10 (sometimes simply referred to as the "technology") is described herein that enables virtual eye-to-eye contact during video interaction between two or more participants 12, 14, 16. This technology is natural, comfortable, intuitive, effective, inexpensive, and easy to use. The technology enables projected eye contact, perceived eye contact, and simultaneous perception and projection of eye contact between participants in a video interaction, and virtual eye-to-eye contact between participants if both participants are using the technology. The technology may also provide additional features and functionalities for enhancing, simplifying, facilitating, accelerating, enriching, and improving the experience of one or more participants in a video interaction.

[0044] The terms "virtual eye-to-eye contact" and "eye-to-eye video interaction" are used broadly to include, for example, the virtual projection of eye contact and the virtual perception of eye contact (each alone or both simultaneously). Virtual eye-to-eye contact can occur in video interactions and typically does not occur in live, face-to-face interactions.

[0045] This technology can be applied to a wide range of video interaction scenarios, including: meetings, video calls, conferences, webinars, medical appointments, parties, social gatherings, courses, podcasts, lectures, seminars, board meetings, discussion groups and combinations thereof, as well as a variety of other video interaction scenarios, some of which will be discussed below. This technology is particularly useful and effective in video interaction scenarios sometimes referred to as video conferencing.

[0046] The terms “eye-to-eye video interaction” and “eye contact video interaction” are used broadly to include, for example, any electronic communication, dialogue or other exchange between two or more participants, wherein such electronic communication, dialogue or other exchange may include virtual projection of eye contact, virtual perception of eye contact, or virtual eye-to-eye contact between the eyes of one participant and the eyes of another participant via a video medium.

[0047] The term "participant" is used broadly to include any one or more people participating in a video interaction, for example, on either end. Two or more people on opposite ends of a video interaction are sometimes referred to as "local participants" and "remote participants." As described below, local and remote participants may be very close to each other, including being in the same room or space. Sometimes, a participant is referred to as a "local participant" because they are a component of the technology that is the main focus of a particular discussion.

[0048] Eye-contact video interaction technology includes one or more computing devices 18, 20, or is supplemented by one or more computing devices 18, 20, which host one or more commercially available or proprietary application-based or web-based video interaction platforms 22, 24 (such as video conferencing platforms). The video interaction platform enables participants to engage in video interaction and control the progress and status of the video interaction, the view being presented, sharing of the participant's desktop, inviting participants, muting and unmute microphones, and various other functions and features, including but not limited to those commonly used during video interaction over the Internet. Furthermore, this technology allows participants to engage in other activities and use other applications 27, 28 on the computing device while distinguishing those activities in their mind from the eye-contact video interaction.

[0049] The term "computing device" is used broadly to include any device that includes, for example, a processor capable of executing instructions to perform processing or tasks or to provide features or functions. Computing devices include computers, workstations, laptop computers, tablets, telephones, mobile phones, voice assistants and mobile devices, and a wide variety of other examples.

[0050] The term "video interaction platform" is used broadly to include any software, application, web-based application, or other facility used to host, present, control, facilitate, collaborate on, or manage one or more video interactions between participants via a communication channel. In some examples, a video interaction platform may acquire and process video captured by a camera, generate video display information and feed it to a video display, present, manage participant interfaces and interact with these interfaces to control the state, configuration, and operation of the video interaction, and perform various other functions associated with providing video interaction to participants. Examples of video interaction platforms may provide video conferencing, security, marketing, information, entertainment, and various other functionalities.

[0051] In some implementations, the computing device is permanently or temporarily integrated into an autonomous, independent eye-contact interaction device 26. In some examples, the computing device is an autonomous, independent device (such as a laptop computer, mobile phone, or other mobile device) coupled to the eye-contact interaction device. Other collaborative arrangements between the eye-contact interaction device and the computing device are also possible.

[0052] In some cases, a computing device receives and processes video capture signals from a camera and generates video display signals for delivery to a video display. A computing device-hosted video interaction platform can manage the processing and use of video capture and display signals to serve the features and functions involved in eye-contact video interaction.

[0053] The term "camera" is used broadly to include any device capable of capturing a series of images or frames of a subject at a rate, such that the rate is high enough, for example, to create for the viewer of the sequence the impression of eye contact, including eye-to-eye contact with the person captured in the images or frames. Examples of cameras may include digital cameras, smartphone cameras, webcams, cameras on mobile devices, and a variety of other devices.

[0054] The term "video display" is used broadly to include any device capable of presenting a series of images or frames of a subject at a rate, such that the rate is high enough to create an impression of eye-to-eye contact between the viewer of the sequence and the person included in the presentation. Examples of video displays include CRT displays, flat panel displays, digital projectors, and others.

[0055] Overview of implementation and use

[0056] For example Figure 1 As shown, in some implementations, the eye contact interaction device 26 includes a video display 30, wherein the video display 30 has a display axis 32 perpendicular to a position 34 (e.g., at the center) on the display surface 36 of the video display.

[0057] The term “display axis” is sometimes used to refer to an imaginary line that extends from the position of the video display (e.g., at the center) and is perpendicular to the plane of the image or frame being presented.

[0058] The eye-contact interaction device includes points 38 and 40, with a video display supported at point 38 (e.g., held in a fixed or adjustable position and orientation) and a camera 42 supported at point 40 (e.g., held in a fixed or adjustable position and orientation). The camera has a video capture axis 44 along which it captures video of the participant 12.

[0059] The eye-contact interaction device includes an axis alignment device 46. The axis alignment device, points 38 and 40, and the orientation of the camera and video display held at these points are arranged such that the display axis and video capture axis point toward the axis alignment device, which then aligns the display axis and video capture axis parallel to each other and (in some implementations) on a common line (also called a common axis) 48. This axis alignment has the effect of enabling eye-to-eye video interaction (including individual or simultaneous projected and perceived eye contact) between a user capable of performing eye-contact interaction (both viewing the display (perceiving eye contact) and being captured by the camera (projecting eye contact)) and other participants using a computer, laptop, mobile phone, other mobile device, or other eye-contact interaction device connected to the local eye-contact interaction device via a communication channel 50 (e.g., via the Internet 51). Assuming remote participants 14 and 16 are watching along the video capture axis 56 of the remote camera 58, for example, if the remote participants are also using an eye-contact interaction device of the kind discussed herein, then the local participant 12 (the person using the local eye-contact interaction device) will have the advantage of eye-to-eye video interaction. In this case, the remote participant will have an eye-to-eye view of the local participant's video presented on the remote video display 60.

[0060] The term "video capture axis" is sometimes used to refer to an imaginary line that extends from the plane of the camera's image sensor and is perpendicular to the center of the camera's field of view.

[0061] In some implementations, the local participant can simply engage in the natural movements of the subject (which can be any kind of visible subject in the video, including the remote participant) watching the eye-contact video interaction (e.g., projecting eye contact onto the subject). This is implied by alignment along a common line from the video display toward the local participant. The local participant is then automatically captured by a camera directly looking at the camera (e.g., projecting eye contact). Thus, the remote participant will perceive that the local participant is directly looking at her (e.g., perceiving eye contact).

[0062] In some instances, such as when a local participant is presenting educational information or communicating with a large number of remote participants, a local eye-contact interaction device will provide eye-contact video interaction with each remote participant. In this sense, the local participant can project eye contact, and each remote participant will perceive that the local participant is looking directly into his or her eyes (e.g., will perceive eye contact). Furthermore, the local participant will understand that he is being seen in that way by the remote participants (in other words, the projection of his eye contact is being perceived as eye contact) and can present his information comfortably, intuitively, and effectively because he will understand that all remote participants are seeing him looking directly into their eyes. When the number of participants is relatively small, say less than 50, less than 25, or less than 5, or simply one local participant and one remote participant, it can also be useful to provide a fully bidirectional eye-to-eye video interaction device to all remote participants.

[0063] In some implementations, the computing device has a processor 62 and a storage device 64 for instructions executable by the processor. In some implementations, the computing device executes one or more video interaction platforms as an application or via a browser page (using instructions). In some implementations, the local video interaction platform operates via a server 66 to communicate with a remote video interaction platform 24 (which may or may not be of the same kind as the local video interaction platform) to enable eye-contact video interaction between local and remote participants. For this purpose, the local video interaction platform may also present a user interface 70 on the display 72 of the computing device, drive the video display to present information and video received from the remote video interaction platform via the communication channel 50, receive video from the camera 42, process the video, and send at least a portion of the video to the remote video interaction platform via the communication channel, etc. The computing device is connected to the video display and camera via cables or wireless channels conforming to standard input and output communication protocols such as HDMI and USB.

[0064] Eye-contact interaction devices can typically be placed at a comfortable height on a desk next to the main computer to facilitate relaxed eye-to-eye communication. For this purpose, eye-contact interaction devices can include stands and bases, as discussed later.

[0065] The local video interaction platform may include computing device 18, and can be operated not only by computing device 18 but also by one or more other computing devices. Two or more such computing devices can collaborate in providing the features and functions of this technology, as well as other features and functions. For example, a smartphone can be used as a second computing device to collaborate with computing device 18 to enable participants to interact using the technology. One or more user interfaces can be presented on any combination of one or more computing devices.

[0066] While the discussion so far has emphasized aspects involving video and visual perception and expression, this technology can include interaction modes, devices, and software associated with other types of perception or expression, or combinations thereof. For example, eye-contact video interaction can be associated with or accompanied by sensory expressions or perceptions enabled by touch sensors or tactile feedback, audio expressions or perceptions via microphones or speakers, gestures, and others.

[0067] Through the user interface 70, local participants can launch the local video interaction platform, log in, control the settings of their operation, view the live video of remote participants, and perform various other tasks.

[0068] One or more computing devices may also initiate and execute other programs in a typical or atypical manner, present a user interface to local participants, and interact with local participants in relation to other programs.

[0069] In various examples, the proportion of the total time spanned by an eye-contact video interaction session can vary depending on the video interaction context, the progress and state of the video interaction, the participant's state and preferences, and other factors. The length of the continuous eye-to-eye contact period can also vary, during which the participant is engaged in direct eye-to-eye contact (including projected or perceived eye-contact) or other activities via the user interface of the computing device or other components of the technology. In some cases, local participants may use eye-contact interaction devices for extended periods of traditional eye-to-eye contact (including projected or perceived eye-contact) interspersed with occasional, simple, routine use of the computing device to control the eye-to-eye interaction platform and interact with the platform and other applications and web-based facilities.

[0070] The features and combinations of features of eye contact interaction technology can be implemented in a wide range of methods, devices, software and systems, some of which are described below as example configurations.

[0071] Example configuration

[0072] Configuration A—An integrated eye-contact interaction device without a computing device

[0073] like Figure 2 As shown, in some implementations, the eye contact interaction device is standalone (in addition to the computing device 97) and includes an integrated video display 72, an integrated beam splitter 74, an integrated camera 76 or a point 78 for receiving added camera devices, and a housing 80 for containing the display, beam splitter, camera and other components in some instances.

[0074] Video shows

[0075] In some cases, integrated video displays are high-resolution, appropriately sized flat-screen displays. The image quality of the video presented on the video display is a crucial factor in the quality, realism, and effectiveness of eye-contact video interactions. Features of the video display used to improve video rendering (including its sharpness, color reproduction, sharpness, resolution, brightness, contrast, color temperature, saturation, and combinations thereof) are useful. One objective is to provide participants with video interactions that are perceived as as realistic and authentic as possible. Therefore, the video presentation will not consciously or subconsciously interfere with the natural and comfortable eye-contact video interactions between participants and other participants (including the virtual projection and virtual perception of eye contact). Participants' confidence in the realism and naturalness of the eye-contact video interactions can enhance the quality of the participant's experience and the effectiveness of the interaction as perceived by both the participant and other participants.

[0076] Integrated displays can have a variety of possible characteristics, including the following in some examples.

[0077] 1. Size. The diagonal dimension of the display surface should not be too large so that the interactive video device does not take up unnecessarily large amounts of table or desk space. Conversely, a size that is too small will impair the realistic impression conveyed to the participant. A good size is in the range of 6 to 10 inches; an especially good size is about 7 inches diagonally across the display surface.

[0078] 2. Resolution. In some examples, because the integrated display will be the primary focus of participants' attention during video interaction, the integrated display's resolution can be high, at least 1080p or 4K. For example, if the display of the computing device is only used infrequently for controlling the integrated display, screen sharing, checking emails, or performing other tasks, then a high resolution is not necessary.

[0079] 3. Brightness and Color Rendering. The integrated display should be bright enough to produce a realistic presentation, as participants will be viewing reflections of video in beam splitters in some cases. In some implementations, the integrated display can have a brightness ranging from 250 to 1000 cd / m². In a particular example, the brightness is 500 cd / m², the contrast ratio is 1000:1, and the viewing angle is 1700 / 1700 (H / V). The integrated display should be able to render video with correct colors.

[0080] 4. High-definition video output. In some cases, the video display may include HDMI or other high-definition video outputs 91 to connect to an external video display 90 (e.g., a larger video display) or to a projector to interactively project video onto a wall for a larger audience. Such connections will allow video to be “mirrored” from the integrated video display to the external video display.

[0081] In some cases, integrated video displays can be the type of monitor accessory used with cameras or other digital webcams, and can have the capability to render high-quality video. One commercially available example is the Lilliput A7S 7-inch camera field-of-view monitor. This monitor supports 4K HDMI input loop-out, has a 1920x1200 resolution, a 1000:1 contrast ratio, and 500 cd / m². 2 Brightness and a 170-degree viewing angle, and it can be viewed from Amazon. https: / / www.amazon.com.au / Lilliput-Monitor-Supports-1920x1200-Resolution / dp / B074X9JTRC / ref=asc_df_B074X9JTRC / ? t ag=googleshopdsk-22&linkCode=df0& hvadid=341791754774&hvpos=&hvnetw=g&hvrand=13271574732989024752&hvpone=& hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9070562&hvtargid= pla-569851224162&psc=1 Obtained from [location].

[0082] The video input to the integrated display (e.g., the Lilliput display identified above) can be an HDMI signal, which is delivered via HDMI cable 94 from the HDMI output 96 of the computing device 97, which may be generating video for display, to the input 92 of the integrated video display. If the computing device does not have an HDMI output, it can be connected via a USB-C to HDMI adapter or cable, or in other ways.

[0083] Power for the display can be supplied by a plug assembly (power adapter) for supplying approximately 12 volts DC at 1 amp, such as the Altronics plug assembly available from Amazon. https: / / www.altron ics.com.au / power-supplies / power-packs / ? type_1=ac-to-dc-power-adapters ).

[0084] Various other integrated displays from Lilliput and other suppliers may also be suitable.

[0085] Among other things, one or more of the following features of an integrated video display (or other point of contact) (which may be pre-assigned to buttons on the front) are useful for good eye contact video interaction.

[0086] 1. An optional (on-off) target (e.g., a red cross) for marking the center of the screen for the participant. The marked center of the screen can be considered as the endpoint of the aforementioned display axis, such that when the participant is looking directly at the red cross along the display axis, the participant is necessarily being video-captured by a camera, with her eyes guided along the video capture axis.

[0087] 2. The ability to horizontally flip display material on a screen so that text reflected from left to right in a beam splitter is readable.

[0088] 3. Adjustable brightness and contrast.

[0089] 4. Selectable aspect ratio and scaling function.

[0090] The size and configuration of eye contact interaction devices can depend primarily on the size and aspect ratio of the beam splitter.

[0091] Axis alignment device (e.g., beam splitter)

[0092] In implementations using planar glass beam splitters, a type can be used that is incorporated into the speech cue and has a 70 / 30 ratio of light transmission (to the camera) to reflection (from the integrated video display). The glass thickness should be chosen to be thick enough to achieve rigidity without being unnecessarily heavy. An example of a suitable glass beam splitter is one that can be cut to size from a Leederville camera. https: / / www.leedervillecameras.com.au / products / accessories / studio-equipment / prompt-it-maxi-teleprompter-k it The types included in the obtained speech prompter suite.

[0093] It can also be obtained from the United States https: / / telepromptermirror.com / glass-teleprompter- mirror / and China https: / / www.alibaba.com / product-detail / Wholesale-Price-2mm- 3mm-3-2mm_62345959902.html Suppliers can obtain suitable large pieces of glass at a lower cost per unit of finished product.

[0094] Camera

[0095] A wide range of cameras can be used, from simple camera sensors housed in a simple container and mounted directly on the housing of the eye-contact interaction device to full-frame professional digital cameras with HDMI output (such as the Sony Alpha 7) mounted on the support frame of the eye-contact interaction device. Cameras with high-level features and functions, as well as high-quality image generation parameters, can provide better quality video and are likely more suitable. High-quality video has the benefit of a potentially better experience for remote participants (if the remote participants have a display that can take advantage of that video quality). Implicitly, local participants can also have a better experience and engage more effectively in eye-contact video interaction by knowing that the quality of the video captured locally and presented remotely is high.

[0096] Cheaper cameras can also be suitable, such as popular mirrorless cameras that use APS-C size sensors (examples include the Sony a51000 and Sony a6000 as relatively low-cost 4K cameras, as well as the Canon M200).

[0097] In some examples, the camera can be a network camera, which is simpler to connect and does not require a separate power supply or interface device. Some network cameras produce high-quality video signals, such as the Logitech StreamCam. https: / / www.logitech.com / en-us / product / streamcam The webcam is capable of outputting video signals at 1080p and 60fps and requires only a single USB-C connection to the computing device.

[0098] The camera may include HDMI or other protocol outputs 101 to provide video signals to a computing device.

[0099] Camera communication interface

[0100] Cameras typically generate and deliver video signals conforming to common communication protocols (at physical or wireless output). Computing devices can typically accept and use video signals conforming to one or more common communication protocols (at physical or wireless input). The camera must be connected to the computing device via an interface 105 capable of carrying video signals, and in some cases, protocol conversion is performed from the output protocol used by the camera to the input protocol used by the computing device, such as USB or Thunderbolt protocols. In some examples, the interface is a cable connecting an HDMI output to an HDMI input.

[0101] In some implementations, the interface allows the camera's HDMI output to be captured by a computing device. (ElgatoCamlink) https: / / www.elgato.com / en / gaming / cam-link-4k This is one example of an interface with HDMI input and USB output for connecting to a computing device. Another example is available from Amazon... https: / / www.amazon.com.au / Razer-Ripsaw-Professional-Grade-Mix-Playstation / dp / B01CVOMZDM The Razor Ripsaw is available from [location omitted]. The Razor Ripsaw features HDMI input and USB output for connecting to computing devices. In addition to the video signal carrying interface, the camera may also need to be connected to an external power supply.

[0102] case

[0103] Various designs are possible for the housing. Among them, the design considerations are as follows.

[0104] 1. When the housing is included in an eye-contact interaction device, the housing can securely hold the camera, video display, and beam splitter in the correct position, orientation, and distance, enabling the establishment and maintenance of alignment between the video capture axis and the video display axis.

[0105] 2. The housing should be constructed to include walls or other features to provide adequate shielding against ambient light reaching the camera lens, the surface of the video display, and the surface of the beam splitter. Otherwise, the quality of the displayed video or captured video, or both, may be compromised.

[0106] 3. Since the eye contact interaction device housed in the housing may move frequently, it is desirable to make the housing as lightweight as possible while maintaining sufficient strength.

[0107] like Figure 3 and Figure 4 As shown, in some implementations, eight parts are used to construct the shell, which can be 3D printed or cut from a single sheet of material using a computer numerical control (CNC) grooving machine. The 3D printer or CNC grooving machine can be controlled based on the shell design using files generated by CCAD / CAM programs (such as Fusion 360, available from AutoCAD). In some instances, the shell can be made of other materials such as bamboo to significantly reduce its weight, which, among other things, allows it to be mounted on more compact, smaller supports (discussed later), such as supports designed for mobile devices.

[0108] The eight components used to construct the housing include a bottom 102, a left side 104, a right side 106 (shown upside down), a top 105, a back 108, a front 110, a frame 112 for video display, and a support 113 for the camera. At least some of these components include matching tabs and slots, and the tabs and slots can be fastened, for example, with adhesive to form the finished housing.

[0109] The front portion 110 includes a rectangular opening 114 large enough to allow a participant to view the entire display reflected in the beam splitter. A smaller rectangular opening 116 in the front portion, below the viewing opening, provides access to buttons or other interface elements for controlling settings of the integrated video display. The rear portion 108 has a rectangular opening 118 to accommodate the lens end of a camera. A camera support 113 provides a horizontal surface on which the camera can be mounted, wherein the camera's video capture axis extends toward and through the center of the beam splitter towards the participant's eye as the participant is viewing, for example, the center of the reflective surface of the video display.

[0110] In some examples, the camera lens projects through a rectangular opening 108. In other cases, the lens is located behind the rectangular opening and does not project through it. A slot 120 in the camera support accommodates bolts or other fasteners that can mate with a socket or other mating feature on the bottom of the camera to securely hold the camera in place. For example, a typical 1 / 4-inch threaded camera bolt (such as those available from Carbatec) can be used. https: / / www.carbatec.com.au / 149pce-knob-jig-pack-1-4 (Obtained bolts, etc.) to install network cameras.

[0111] The rectangular opening 118 and the camera support with slot 120 are configured to allow for the use of a wide variety of cameras of different sizes, shapes, weights, types, and configurations. For example, since smartphones differ from digital cameras in shape and structure, it is envisioned that the housing can be adapted to allow different cameras and smartphones to be fitted with lenses of appropriate orientation and position. Adjustable elements or modular components of the housing can be used for this purpose.

[0112] The housing has 45-degree slots 122 on both the left and right sides to receive one of the opposite sides of the beam splitter when the housing is assembled. (The 45-degree slots are visible only on one side). In some implementations, the beam splitter slots may be at an angle other than 45 degrees, where the orientation of the display relative to the beam splitter is adjusted accordingly. One of the left or right sides has a rectangular slot 124 for accessing the HDMI input and DC power input of the integrated video display.

[0113] The display frame 112 includes a rectangular window 125 at least as large as the display surface of the video display. The bottom of the frame and housing defines a chamber 126 for housing a video display with a display surface facing upward toward the beam splitter.

[0114] Among other things, the top helps prevent ambient light from reflecting off the beam splitter surface facing the camera.

[0115] In some implementations, the eight components are formed from a single 900mm by 600mm sheet of high-quality AA hardwood plywood using a CNC grooving machine; however, acrylic sheets or other materials or combinations thereof can be used. In some cases, these components or combinations thereof, or almost the entire housing, can be injection molded in a variety of different forms and shapes.

[0116] As will be described in more detail later, in some embodiments, the video display is attached to the beam splitter via a hinge or other mechanism, allowing the video display to pivot against the beam splitter or otherwise move and turn off. When the video display is off, it can be held in place mechanically against the beam splitter, or a magnet can be used to shield the beam splitter and the camera behind it. In some cases, built-in LEDs or other lights can be mounted to extend from the bottom surface of the housing and can be illuminated when off. This arrangement provides a second mode of eye-contact interaction, which can be used, for example, as a desktop lamp.

[0117] like Figure 4 As shown, a 3D drawing of the finished casing without a top reveals the interior and how the beam splitter glass is held in the grooves formed in the side panels.

[0118] Support components

[0119] In normal use, the housing can be mounted or held in a position and orientation that aligns the common axis shared by the video capture axis and the display axis with the participant's gaze axis. To facilitate such a position and orientation, a support 128 can be provided, taking into account the participant's position and orientation as well as the direction she is facing.

[0120] In some implementations, the support is contained within or integrated with the housing. In some examples, the support is separate from and attached to the housing. The connection can be permanent or temporary, allowing the support to be repeatedly attached to and removed from the housing. In some instances, the support can be a tripod, such as a desktop tripod. In some instances, a 1 / 4" adapter for receiving a typical 1 / 4" camera mounting bolt can be included in the base of the housing for attachment to a tripod. Various tripods can be used, such as those available from Amazon. https: / / www.amazon.com.au / Neewer- Centimeters-Portable-Camcorder-kilograms / dp / B01MSAEZN8 The NEEWER tripod can be obtained from [location], or from Leederville Camera. https: / / www.google.com / shopping / product / 14759902322478587975? prds=oid:935888807019972978&hl=en&lsf=seller: 10160362,store:2291968430623775521,s:h Tripods, etc., obtained from [location / location].

[0121] A wide variety of other types of supports can be used to hold the casing in the proper position and orientation relative to the desk, table, or floor.

[0122] The housing and other implementations of the eye-contact interaction device described above offer flexible and adaptable features for supporting cameras, video displays, and other components. Therefore, a wide variety of other configurations, including the following, are possible.

[0123] Configuration B - Replace integrated video display with mobile device display

[0124] like Figure 5 As shown in Configuration A, this configuration uses a housing 140, a beam splitter 142, and an external camera 144 mounted on a support, wherein the lens of the external camera 144 faces the back of the beam splitter. For purposes other than maintaining an interactive session, participants can use a separate computing device 145, separate from the housing. The display 146 of a mobile phone or other mobile device 148 serves as a video display and includes a processor 147, which executes the video interaction platform 149 as an application running on the mobile device or as an application via a web browser service.

[0125] In this configuration, the local participant uses a mobile device to establish and initiate interactive sessions with one or more remote participants. Once a session is established, the local participant places the mobile device into the bottom chamber of housing 140, with the mobile device's display 146 facing upwards. In this configuration, the mobile device display replaces the previously described integrated video display. The external camera continues to function as the camera, and for this purpose, the external camera can be connected to the mobile device via a cable or adapter 154 (such as a cable with one end for the Apple Lightning connector on the iPhone and the other end for the USB connector on the camera). In some cases, an additional application can be installed on the mobile device to replace the mobile device's internal camera with an external camera.

[0126] In practice, this configuration will enable the physical and conceptual separation of the camera used for video interaction from the video display of the mobile device, allowing the display axis of the mobile display device and the video capture axis of the integrated camera to be aligned on a common axis using a beam splitter.

[0127] In this configuration, the local participant can view whatever is being presented on the mobile device's display. The presentation may include video from one or more remote participants, controls for a video interaction platform, and windows from other applications. To control and interact with the mobile device's touch-sensitive surface, the local participant can remove the mobile device from the housing or touch the surface through a window located on the housing frame above the housing.

[0128] Configuration C - Mirrors the video display of a separate mobile device to a video display housed within the housing.

[0129] In some implementations, such as Figure 6 As shown, the configuration may include the aforementioned housing 160, external camera 162, integrated video display 164, and beam splitter 166. A mobile phone or other mobile device 168, separate from the housing, can be used as a computing device. The video interaction platform 170 can be executed on the mobile device, enabling local participants to control eye-contact video interaction, for example, by touching the surface of the mobile device. (See also...) Figure 13 While the mobile device is used to initiate video interaction, a display signal generated by the mobile device for its own embedded display is mirrored to the integrated video display within the housing using, for example, a cable or adapter 172, one end of which may serve an Apple Lightning connector and the other end of which may serve an HDMI connector for the integrated video display. As in configuration B, additional applications can be installed on the mobile device to replace its internal camera with an external camera.

[0130] In configuration C, the mobile device is not placed in a housing; therefore, the interaction between the participant and the surface of the mobile device is less cumbersome than in configuration B.

[0131] Configuration D - Replace the camera held in the housing with a mobile device camera.

[0132] like Figure 7 As shown, in some implementations, the configuration may include a housing 180, an integrated video display 182, and a beam splitter 184, as previously described. In this configuration D, participants establish and initiate interactive sessions via a video interaction platform, and then place a mobile device on a camera support, wherein the lens axis of the mobile device's camera points towards the center of the beam splitter and is at a 45-degree angle to the surface of the beam splitter. Thus, the mobile device's camera replaces the external camera described in configuration A.

[0133] In this configuration, the camera support of the housing can be designed to easily mount and dismount the moving device, and can include adjustable features to enable mounting and dismounting of a wide variety of moving devices.

[0134] As previously described, a display signal generated by the mobile device for its own embedded display can be mirrored to an integrated video display within the housing using, for example, a cable or adapter, wherein one end of the cable or adapter can serve an Apple Lightning connector and the other end can serve an HDMI connector for the integrated video display. Additional applications can be installed on the mobile device to replace its internal camera with an external camera.

[0135] Because the mobile device is located on the back of the housing, it is not easy to access the applications running on the mobile device or the physical controls of the settings. However, remote control can be achieved using, for example, a Bluetooth mouse 186 or keyboard 188 paired with the mobile device, or both.

[0136] One or more microphones or speakers on a mobile device may also be partially obstructed due to their location. Audio can be enhanced using a Bluetooth speaker 194 or microphone 196, or both, as a built-in microphone or speaker on the mobile device, or alternatives to both.

[0137] Configure the E-Mobile device camera as a camera for a standalone computing device.

[0138] This configuration utilizes the integrated video display and glass beam splitter described for configuration A. As in configuration D, the mobile device is placed on a support at the back of the housing and replaces the external camera. Cables or adapters (such as USB adapters) connect the mobile device to the computing device 198 on which the video interaction platform is running. Additional software running on the computing device (e.g., iVCAM) https: / / www.e2esoft.com / ivcam / You can select a camera from a mobile device to use as an external webcam relative to a computing device.

[0139] One of the advantages of Configuration E is that the quality of the video and audio components of a typical mobile device can be high. Configuration E can use the participants' own mobile devices, reducing the cost of the technology. Furthermore, because the video interaction platform is running on a separate (integrated or external) computing device, there is no need for interaction with the mobile device's screen, which might be necessary when the mobile device is used as both a camera and a computing device for eye-contact video interaction.

[0140] Configuration F—Embedding a small form factor computing device within a housing to create a standalone video interaction device.

[0141] In some instances, such as Figure 8As shown, configuration F uses a housing (as described in configurations A and D), an integrated video display, a beam splitter, and an external camera, or a camera from a mobile device mounted on a support of the housing. A small computing device (such as an Intel NUC 200) is integrated with the housing by being mounted in, on, or near the housing. An external camera or camera from a mobile device is connected to the integrated small computing device via a cable or adapter 202. In some cases, the computing device may be a wireless or cable-connected smartphone. The housing's integrated video display is connected to the integrated computing device via a cable or adapter 204 and used as its display. A user interface device 206 (e.g., a mouse or keyboard, or both) is coupled to the integrated computing device via a cable or adapter (such as USB) or via Bluetooth 208 to enable computer control. A video interaction platform is implemented on the integrated computing device.

[0142] The combination of interconnecting components in configuration F provides a standalone, compact, and easy-to-use integrated video interaction device, which can provide eye-contact video interaction to serve any video interaction platform running on the integrated computer.

[0143] In some examples, a fully integrated eye-contact interaction device can use a small computing unit with a 10th-generation Intel i3 processor, 8GB of RAM, and a 250GB NVMe solid-state drive. For example, the processor can run Windows 10 and ZOOM video conferencing software. A wireless keyboard and mouse can communicate with the integrated computer (https: / / www.altronics.com.au / p / a0981-2.4ghz-wireless-media-centre-keyboard-with-trackpad / ). Applications running on mobile devices, such as Mouse Pro on an iPhone, can be used to control the small integrated computer (NUC). Effective operation of the ZOOM platform can be achieved using an external camera, such as in configuration A, or a camera from a mobile device, such as in configuration D or configuration E.

[0144] Configure G-full features.

[0145] The configuration G is designed to include a set of components configured to provide a set of features and functions, which together enable eye contact interaction sessions (e.g., virtual meetings) between two or more participants without involving any other components.

[0146] like Figure 14As shown, in configuration G, the eye-contact interaction device 400 may include two main components: (a) a housing 402 that holds the camera, video display, and beam splitter as previously described for other configurations; and (b) a support frame and base 404 for the housing, such that the height 405 and orientation 407 (e.g., pitch) of the housing can be adjusted relative to a desk or other surface on which the frame and base 409 are placed. In some examples, power, wiring, and other connections within the housing to the video display and camera extend via the frame to a small computing device 404 and connector 406 located in the base.

[0147] The computing device within the base will have WiFi capability and built-in 4G / 5G LTE cellular communication capability for independent internet connectivity, based on the included SIM card 408. The computing device will include an operating system 409 and software 410 for initiating, managing, and terminating eye-contact interaction sessions. The computing device will directly boot into a user interface 412 displayed on an embedded video display, whereby participants can select a video interaction platform such as Skype for the session, choose the microphone, speaker, and video display to use, and determine whether they are components included within the base or external components.

[0148] The base or stand will also include some or all of the following components: (a) one or more high-quality compact speakers 414; (b) a high-quality audio amplifier 416 for driving the speakers with amplified sound from the computing device within the base; (c) an external power supply connector 418 for both the computing device and the video display contained within the housing; (d) a compact keyboard 420 with a touchpad or mouse connector 422 for controlling the computing device via a user interface displayed on the video display (the keyboard and mouse 422 can be detached from the eye-contact interaction device and communicate with the computing device via Bluetooth); (e) (a) An integrated high-quality microphone 426 within the base or stand and an input 428 to an external microphone; (b) a connector 430 (e.g., an HDMI switch) for accepting input from an external computing device 432, such that the video display in the housing is driven by the external computing device rather than by the integrated computing device within the base; (c) a connector 434 for connecting the camera within the housing to an external computing device 433, rather than to the integrated computing device within the base (therefore, the eye-contact interaction device can be used to host sessions from its own built-in computing device or from an external computing device, and the connector within the base allows either option to be enabled).

[0149] As described later, the eye-contact interaction device can be modularly constructed, allowing the housing containing the camera, beam splitter, and video display to be attached to various different stands, such as the full-featured stands and bases described above, or simple stands and bases that provide support and otherwise have limited functionality beyond the connectors required for power, the camera, the video display, and possibly speakers and audio amplifiers. In configuration G, the eye-contact interaction device includes a USB-C hub 440 with an HDMI output 442. This allows a single connector from the computing device to receive the output from the built-in camera and transmit video of the eye-contact interaction session from the computing device to the video display for reflection in the beam splitter (with a transmittance / reflectance ratio of approximately 70 / 30). The hub's HDMI output connects to the device's built-in video display.

[0150] The eye-contact interaction device includes an HDMI output from which video signals can be sent to a connected, optional, larger video display, allowing others in the local space to watch the remote participant.

[0151] The eye contact interaction device may also include an audio interface 470, an amplifier 472, a speaker 474, and an optional headphone output 476.

[0152] The volume of the audio portion of the video can be modified by the volume control 478, and the device can also have a hardware mute switch 480 for the microphone 482.

[0153] Most audio interfaces are based on the Texas Instruments TI 270x chip shown in the figure, which provides similar functionality to audio interfaces such as the Rode A1 USB interface.

[0154] USB microphones such as the Rode NT USB mini can be plugged directly into an integrated USB-C hub, but ideally, eye-contact interaction devices will also have an XLR input 484 with phantom power 486 for an alternative microphone 488.

[0155] In some implementations, the option to use a separate computing device is not built into the base of the eye-contact interaction device in order to simplify construction and design and reduce costs.

[0156] In some examples, the base may have a recess for an optional internal Intel NUC or other small computing device; then, the input of the integrated USB-C hub can be connected to the internal small computing device instead of an external computing device. In such a case, video signals can be routed to the internal computing device and fed from the internal computing device to the video display via the hub. In some implementations, the NUC or other small computing device may be housed within a separate, custom-designed additional base assembly. The separate base assembly may have a recess or other matching arrangement on its top, and the device may have a protrusion or other matching arrangement on the bottom of its base, wherein the protrusion or other matching arrangement adapts to or otherwise mates with the recess in the top of the custom-designed additional base assembly. The power supply 492 for the small computing device may be independent or derived from an eye-contact interaction device.

[0157] Typically, such custom optional additional base components can be separate from and distinct from the device and its base, can be distributed independently, and can contain various elements that add functionality to the device if required by the user.

[0158] like Figure 19 , 20 As shown in Figure 21, in some implementations, it can be added as an upgrade by adding an Intel NUC or other small computer as an intermediate component. For example, the small computer can be housed within a recess of a transparent plastic frame that engages (e.g., mates) with the top of the base. This method has at least the following advantages: (1) For users who will be using their own external PC or laptop computer as their computing device (which is probably the majority of users), the appearance of the device is not affected when the transparent plastic frame is not in place. (2) The frame can be repeatedly added and removed without modifying the base and without fasteners or screws hitting the device. (3) Since the base of the elegant device is transparent, the beauty and details of the base of the elegant device are not obscured; the frame with the small computer almost appears to float on top of the base.

[0159] In some cases, such as Figure 22 As schematically shown, a modular rear camera housing 802 can be provided for the interactive device 804 (see the section on modular features below). In some instances, the modular rear camera housing 802 can be configured to retain a specific, affordable OEM camera 803 as a standard option. In some instances, it may be difficult to find an OEM source to supply a relatively inexpensive, high-quality camera with, for example, a high-quality ASP-C configuration sensor.

[0160] In some examples, as a modular option, the interactive device may include a replaceable camera housing 806 configured for one or more commercial cameras (e.g., mirrorless cameras manufactured by Sony, Nikon, or Canon). This allows users of the interactive device to purchase a higher-quality camera as an upgrade to the camera setup by simply replacing the modular rear camera housing 802 with an optional camera housing 806 designed for commercial cameras. Different optional rear camera housings can be used for the corresponding commercial cameras. The support assembly 808 of such a rear camera housing may have adjustable y-axis and z-axis positions, allowing the support assembly 808 to support different models with lenses aligned with the video capture axis. In some implementations, to simplify the shielding of external light by the lenses and beam splitters, separate custom camera housings 810 can be provided for each different camera. Such housings are less complex and can be custom injection molded for the corresponding camera model. The rear of the interactive device may have a power supply and an HDMI / USB socket, allowing the camera to be connected to appropriate cables for power and video signals. Because the power requirements of these cameras can be varied, the DC power can be adjusted using a miniature LCD display that is set to show the selected voltage.

[0161] The internal computing device can be controlled via USB (or preferably Bluetooth) connection to an external compact keyboard with an integrated mouse pad 494.

[0162] although Figure 14 The components shown are divided in a specific way among the housing, support frame, and base; however, depending on the design and configuration, any one or any combination of two or more of these components may be included in any combination of the housing, support frame, and base. Furthermore, although the housing, support frame, and base are shown as distinct components in a specific order or configuration, they may be organized in different orders or configurations, or may be combined or further subdivided into one or more components. For example, the housing, frame, and base may be combined into a single eye-contact interaction device, which, in use, may be mounted or placed on any other surface or device, or held as a portable unit.

[0163] Configure H-embedded tablet PCs as video displays and computing devices

[0164] like Figure 15As shown, in this configuration H, a mobile device 500, such as a tablet computer, can be incorporated into the housing 502 of the eye-contact interaction device and can replace the video display, serving as a computing device running a video interaction platform (such as ZOOM) for hosting eye-contact video interaction sessions. In some implementations of such a configuration, a camera 504 in the housing is inserted into a USB-C connector 506 in the tablet computer to replace the tablet computer's built-in camera and maintain eye-to-eye contact. The housing is configured to allow insertion of a tablet computer to replace the built-in video screen, and the housing is supported by a stand 507 and a base 508 or tripod.

[0165] Most tablet computers include a microphone and a speaker. An external microphone 510, an audio amplifier 512, and a speaker 514, contained within the base of the eye-contact interaction device, can be attached via a USB hub 516 that is built into the base and attached to the tablet computer's USB-C connector 506.

[0166] Tablet computers typically have WiFi or built-in 4G SIM card technology 520 for independent internet connectivity 522. This technology can be used to connect eye-contact interactive devices to the internet.

[0167] A separate keyboard 524 that communicates using Bluetooth can be contained within the base or external to the base.

[0168] Eye-contact interaction devices designed to include a tablet in this configuration can be cheaper than the full-feature configuration G described above, which may include the cost of a dedicated, suitable tablet. Not all tablets will be suitable, and for some mobile devices such as iPads running iOS, connecting a camera as an external source may not be straightforward. Windows-based or Android-based tablets, such as the Windows 10 Microsoft Surface Go 2, may be more appropriate.

[0169] An alternative to providing participants with tablets in configuration H would be to incorporate a suitable tablet-type device into the housing, as an alternative to having a separate video display and an integrated computing device within the base. As mentioned above, the advantage of having a separate video display and a separate computing device within the base is that these individual components can be specified and customized individually as required. For example, an eye-contact interaction device with a keyboard or touchpad within the base would not require a touchscreen.

[0170] In some instances, a design can serve as a low-cost eye-contact interaction device requiring an external computer, or as a full-featured device that includes a 4G SIM card, enabling internet connectivity even without Wi-Fi. The keyboard and touchpad can be detachable and mounted into a base that may include a speaker, a connector for microphone input, and other features as described above.

[0171] Use of eye contact interactive devices

[0172] In some usage patterns, to operate, control, and interact with an eye-contact interaction device, participants can download and install a mobile application on a mobile device (such as a smartphone). This mobile application enables the initiation and maintenance of a wireless connection (e.g., a Bluetooth connection) between the mobile device and the eye-contact interaction device. The application can display menus on the mobile device's screen, allowing participants to interact with and control the eye-contact interaction device by navigating through the menus. Signals generated as a result of user interaction can be applied to an application running on the computing device of the eye-contact interaction device. Therefore, in some examples, the mobile device is, in a sense, used as a remote control of the eye-contact interaction device.

[0173] In some implementations, participants can interact with and control the eye-contact interaction device using, for example, a Bluetooth mouse or keyboard, or both, a computing device wirelessly connected directly to the device. In some examples, participant interaction can be achieved through a user interface displayed on the video display of the eye-contact interaction device.

[0174] Based on the use of an application running on a computing device on a mobile device or in an eye-contact interaction device, a participant may initiate, perform, manage, and terminate, for example, the following functions and combinations of two or more of them: (a) turning the eye-contact interaction device on or off; (b) pairing a mobile device or wireless mouse and keyboard with different eye-contact interaction devices using Bluetooth or other wireless connections or cables; (c) registering an eye-contact interaction device with one or more participants (by linking one or more participants' accounts to the device); (d) connecting the eye-contact interaction device to the Internet; (e) logging into an account; (f) navigating, controlling, and turning on or off any registered eye-contact interaction devices linked to the main participant's account; (g) upgrading firmware or software; (h) toggling the mute function on or off; (g) switching... This includes availability status among available, absent, and unavailable options; (i) adjusting language and regional settings; (j) adjusting audio and visual settings (e.g., the intensity of light projected onto participants), adjusting camera zoom settings and focus range, adjusting the brightness and color settings of the video display, volume, microphone sensitivity, or whether to use a screen saver; (k) adjusting the position, location, and orientation of eye-contact interaction devices (e.g., if a servo motor is installed, the unit can adjust its orientation or height); (l) assigning and changing the function of one or more physical buttons (e.g., making a physical button a "mute on / mute off" switch or changing the function of the button to readjust the rotation, zoom in and out, or aspect ratio of the video display); (m) navigating the "App Store"; and (n) installing, logging in, managing, initiating, and terminating one or more telecommunications applications and services.

[0175] The settings of the built-in camera (e.g., focus, exposure, resolution, white balance, zoom, and others, and combinations thereof) can be set in a variety of ways. These ways include: (1) software running within the computing device, (2) using the camera or a physical touchscreen or mechanical switch on the back of the device, and (3) remote use via wireless or Bluetooth communication with an app on a smartphone.

[0176] Telecommunications applications and services may include, for example, one or a combination of two or more of the following: (a) Zoom; (b) Microsoft Teams; (c) Google Meet; (d) Slack; (e) Skype; (f) WhatsApp; (g) FaceTime; (h) Messenger; and (i) others. Therefore, eye contact interaction devices are agnostic to video interaction platforms.

[0177] In a simple use case, a participant can use her smartphone to navigate menus and initiate a Zoom session on a nearby “paired” eye-contact interactive device, then invite other participants to join the session.

[0178] Account and device registration

[0179] Typically, participants can create different participant accounts for each of the two or more eye-contact interaction devices themselves using an application running on a computing device. In some cases, accounts can link specific eye-contact interaction devices to a video interaction platform. For example, one account could be identified as "Michael's eye-contact interaction device at home on Zoom," and another account could be identified as "Michael's eye-contact interaction device at his office on Teams." This account technology allows one or more eye-contact interaction devices to be added or removed as participants in one or more video interaction sessions on a video interaction platform. This arrangement allows participants to log in to a video interaction session from their desktop computer (e.g., to share content from their desktop computer screen) and also to log in to the same video interaction session via their eye-contact interaction device—both of which can serve as the primary channel for eye-contact interaction.

[0180] For example, registering a specific account for a given eye-contact interaction device also enables a participant to remotely manage more than one eye-contact interaction device, for example, using software controls (using a mobile app or an application running on one of her registered eye-contact interaction devices) as described above. This allows the participant to remotely log in to different eye-contact interaction devices. For example, a participant at home can use one eye-contact interaction device at home to operate and activate other eye-contact interaction devices located in her office. With this arrangement, the participant can "appear" in a remote office by appearing as an "avatar" on the office eye-contact interaction device. For example, imagine a CEO working from home using her registered home-based eye-contact interaction device. She can dial in (i.e., connect) to a registered office-based eye-contact interaction device from her home-based device and keep office hours open for any worker physically near the office location or her registered office-based eye-contact interaction device. In some implementations, the method can be extended to enable multiple eye-contact interaction device participants to register and access (connect to) multiple eye-contact interaction devices. In the example above, other members of the execution team were also able to connect to the "office-based eye contact interaction device" mentioned in the example.

[0181] Multiple eye contact interactive devices

[0182] The use of eye-contact interaction devices is characterized by the ability to connect a given eye-contact interaction device to one or more other eye-contact interaction devices. For example, a single eye-contact interaction device participant can simultaneously operate and use more than one eye-contact interaction device, thus having, for example, two separate, parallel eye-contact video interactions. One eye-contact video interaction session can be conducted from a first eye-contact interaction device and involve two or more groups of remote participants. A second, simultaneous eye-contact video interaction session can be conducted from a second eye-contact interaction device and with different groups of two or more remote participants. For example, imagine a single executive meeting occurring between multiple remote executors. An executor can also have a separate eye-contact interaction device that operates in an "always-on" manner during an interaction session with their executive assistant, allowing them to transmit instructions to their executive assistant in a confidential manner relative to other participants during the interaction session. For example, an executor can briefly mute all other sessions except their session with their executive assistant to provide instructions or make requests.

[0183] Preset

[0184] Through mobile app or computing device application, participants can create preset shortcuts for running predefined workflows. For example, with the click of a single button, an eye-contact interactive device can log in to a preferred video interaction platform (e.g., Zoom), adjust lighting and sound quality to a predefined set of values ​​(e.g., high brightness), and call a specific set of remote participants (e.g., an execution team).

[0185] Other applications

[0186] Participants may also install or remove additional mobile or computing apps specifically built for the use of eye-contact interaction devices. For example, a standalone remote health app can overlay image processing onto the video and audio signals output from a computing or mobile device to detect physiological measurements (e.g., interpreting subtle pixelated changes in the video feed at 30 frames per second and interpreting that signal as heart rate). The user interface of such an application may also optionally display information to facilitate video conferencing, including one or more of the following: the current time and time in any number of other pre-selected time zones; the time elapsed since the start of the eye-contact video interaction session; the speed or signal strength of the internet connection at each end of the session connection; and the number of previously connected participants currently available.

[0187] In the implementation (where a camera function is provided by the participant's own mobile device, for example, attached to the back of the eye-contact interaction device housing, and the display signal of the mobile device is mirrored to the video display of the eye-contact interaction device), the participant can initiate an eye-contact interaction session via a dedicated mobile device application, which simplifies the establishment of the participant's session regardless of the video interaction platform used. For example, the participant might click "Zoom Call" to cause the display to automatically configure itself to optimize the participant's experience through the eye-contact interaction device.

[0188] Other features and considerations

[0189] Support components

[0190] In some implementations, such as Figure 9 As shown and described above, the integrated support 210 can be attached to the bottom of the housing or formed as part of the housing. The support may include an integrated speaker 212 and an integrated microphone 214, and includes the capability for Bluetooth or wired connections 216 or both with a computing device 218 or a mobile device 220. In some cases, connections to the computing device or mobile device can be made directly through the support within the housing. The support may also have a socket 222 for accepting a USB connector for a separate high-quality microphone 224, replacing the built-in microphone of the support. An external speaker 226 may also be attached to the support as a replacement for the built-in speaker. Additionally, as required by a video display, camera, mobile device, or small computing device, or a combination thereof, the support can function as a hub providing DC voltage via a DC socket 228 or a USB port 230. This allows a single power connection to the support to serve all components.

[0191] In some cases, the support may include an integrated minicomputer 234 that will initiate and perform the video interactions as described above.

[0192] power distribution

[0193] In some implementations, displays require 7 to 12 volts, and mobile devices such as iPhones require 5 volts. To meet these needs, a single 9-volt DC power converter can be attached to a step-down transformer (e.g., https: / / www.altronics.com.au / p / z6334-dc-dc-buck-module-3-40v-input / A single 9-volt DC power converter is housed within an inline 3D miniature housing. A cable with a 2.1mm DC plug carrying the 9-volt DC power can supply power to the video display. A 5-volt DC cable terminating at an Apple Lightning connector can power mobile devices. Therefore, the configuration of a video interaction device using a mobile phone or other mobile device can be standalone, capable of initiating eye-touch video interaction using a video interaction platform, and will have only a single power cable for all components. The support structure can be configured to include similar power distribution circuitry.

[0194] Projector replacement

[0195] In some implementations, compact LED or DLP projectors can be used as replacements for built-in video displays. Typically, such projectors may not be as bright as LCD, LED, or IPS flat-panel displays. Using a projector will also require a semi-reflective screen in front of the housing capable of projecting the image, which may interfere with video capture by the camera. Projectors can also be used as replacements for integrated video displays to project the display onto a screen in the location of the integrated video display.

[0196] Display magnification

[0197] In some cases, using a magnifying lens to enhance eye contact with interactive devices for video display can improve the participant experience. For example, a 2x Fresnel lens can be placed in front of the video display surface. This method can cause reflections on the video display and interfere with the camera's focusing function and image quality; these problems can be addressed using higher quality lenses and camera focusing strategies.

[0198] Modular

[0199] In some examples, such as Figure 10 As shown, eye-contact interaction devices can be designed and marketed as modular systems 250, which may include or exclude and later add or remove various available components 252 and various models or types of a given component to produce device instances 254, 256, all of which are derived using the same supplements to the available components. Therefore, a wide selection of possible instances can be presented to the consumer, who can easily modify the supplements to the components of a given instance of the modular system, thus reducing costs. In addition to the choice of which component supplement to include in the modular system, consumers can also choose between different products to provide the functionality of a given component. For example, if a consumer chooses to include a camera in the modular system, a choice of three (or more) different cameras can be provided to provide the functionality of the camera component. In some cases, three different video device configurations and three different support components can be provided, enabling nine different products to be offered from the modular system.

[0200] The term "component" is used broadly to include, for example, one or more of the following components and combinations thereof: video display 258, axis alignment device 260, camera 262, computing device 264, miniature computing device 266, mobile device 268, support 270, housing 272, microphone 274, speaker 276, communication hub 278, etc.

[0201] In such a modular system, the support may include, for example, (a) a power hub feature only, (b) a power hub feature and one or more of an embedded Bluetooth-enabled speaker and a Bluetooth-enabled support, or (c) a power hub feature having such a speaker and microphone as well as an integrated minicomputer.

[0202] Furthermore, in such a modular system, among a variety of combinations of components, the housing may include an integrated video display designed for use with an iPhone or other mobile device, an integrated video display and an integrated high-end camera, or an integrated video display designed for a mirrorless camera such as the Canon M200.

[0203] Video interactive devices can include a variety of housings that can be combined with various support elements for maximum flexibility. In some instances, limiting the support element and housing options to two may be particularly practical.

[0204] A wide variety of other modular features and configurations will also be possible.

[0205] Control video interaction

[0206] The user interface presented by a video interaction platform for controlling eye-contact interaction devices may include buttons, menus, dialog boxes, and other interface controls. These controls are clicked or information is entered by touching the surface of the display, typing, or manipulating a mouse or other cursor control device. The video interaction platform can be controlled in a variety of ways, including the following technologies and combinations thereof:

[0207] 1. A computing device can be controlled using connected devices such as a keyboard, mouse, touchpad, or combinations thereof, and can be wirelessly controlled via communication over Bluetooth or other wireless connections using a wireless controller.

[0208] 2. Eye-contact interaction devices may include components for controlling a video interaction platform via gestures. Gestures may be expressed by, for example, head, hand, or finger movements or orientations performed "in the air." Finger movements may be expressed by tapping on a display or other touch-sensitive surface. In some embodiments, the eye-contact interaction device may have a touchscreen or touchpad on one side (e.g., top, either side, or bottom) such that finger gestures and swipes can control the eye-contact interaction device and all its functions. For example, waving a hand can turn the eye-contact interaction device on or off.

[0209] 3. User interface control can be achieved using voice or speech interpreted by a computing device or mobile device.

[0210] Access modes and usage modes

[0211] For example, participants may access and control interactive devices using eye contact, as described above, through direct input to a user interface presented on or by a computing device. Other access modes are also possible, including one or more of the following modes.

[0212] In some instances, a device located away from the eye-contact interaction device (such as a computer, laptop, or mobile device) can control the eye-contact interaction device and also control the eye-contact video interaction supported by it. In effect, this arrangement extends the distance between the eye-contact interaction device and the participant controlling its operation. The participant controlling the eye-contact interaction device can be a different person from the participant engaging in the eye-contact video interaction at the device.

[0213] As an example, the local participant could be the patient at home, and the remote participant could be the patient's doctor working from home. Both could use an eye-contact interaction device. Even when physically away from the patient, the doctor could control the eye-contact interaction device in the patient's home, including the ability to turn the device on and off, zoom in and out of the camera on the patient's device, and various other control activities. These control activities could be expressed by the doctor through an interface on, for example, a computer device in the doctor's home, and could be transmitted via the Internet to the computer device in the patient's home for execution.

[0214] For example, control of an eye-contact interactive device from one location to another can be achieved remotely via the Internet or mobile broadband. Such an arrangement is useful for accessing eye-contact interactive devices placed at a distance from the local participant. For instance, a local participant can control an eye-contact interactive device located in their office from their home. In effect, the local participant can "casually visit" a remote eye-contact interactive device and control its operation as if they were physically local to it, without having to be physically near it. In some cases, this can be achieved by communicating signals from a camera in the local eye-contact interactive device to the eye-contact interactive device in the office, and communicating signals from the video display of the eye-contact interactive device in the office to the local eye-contact interactive device. In effect, the eye-contact interactive device at home becomes an extension of the eye-contact interactive device in the office.

[0215] This technology can be applied by participants to control one or more eye-contact interaction devices. In this way, a local participant can individually control their eye-to-eye video interaction with each of several remote participants, and do so one by one, unlike some other implementations where the local participant's eye-contact video interaction is hosted in a substantially identical manner for all remote participants. Among other things, this technology will enable a local participant to use a single eye-contact interaction device to simultaneously support individual, independent eye-contact video interactions with individuals or groups during the course of an interaction session. Such an implementation can be achieved by creating new software for the video interaction platform or enhancing existing software and by enhancing the eye-contact interaction device.

[0216] In some cases, the simultaneous use of multiple independent eye-contact interaction devices will facilitate this technology and enable multiple parallel eye-contact video interactions with options for connecting participants from different groups in different synchronous dialogue networks. Facilitating video conferencing using discrete, independent eye-contact interaction devices at a given location will enable new and unique dialogue arrangements.

[0217] In some examples, using multiple local eye-contact interaction devices, each with overall control (e.g., using built-in computing devices and communication software for each device) connected to different remote participants, in different parallel video interactions will support a virtual workplace similar to a group in a shared physical workplace. The image of the local participant sent to the remote participant will essentially be an avatar of the local participant. A local participant with multiple eye-contact interaction devices on their desktop can communicate individually with the individual avatars of the remote participants or participate in a conversation group. If the camera of one or more eye-contact interaction devices has a field of view that includes the display of one or more other eye-contact interaction devices, the different avatars appearing on the displays of the multiple eye-contact interaction devices can also communicate with each other. In effect, two or more eye-contact interaction devices on a local participant's desk can provide "tunneling" capabilities between two or more remote participants, allowing remote participants to have eye-contact interactions with each other without involving the local participants. Similarly, groups of participants can have separate, different eye-contact video interactions between themselves.

[0218] like Figure 11 As shown, for example, among five participants 300, 302, 304, 306, and 308, four participants at four points 310, 312, 314, and 316 each have three eye-contact interaction devices (depicted as cameras), enabling local participants at each point to have eye-contact video interaction with the indicated other participants. In the figure, participant 300 may, for example, have eye-contact interaction with participants 302 and 304, and participant 302 may have eye-contact video interaction with participants 304 and 306. Then, also as... Figure 12 As shown, the five participants could have eye contact interaction in roughly the same way they would with colleagues who typically share open office plans and with participant 308 being participant 308's executive assistant.

[0219] Another access mode using eye-contact interactive devices can be located within a private, highly secure network.

[0220] While many of the examples discussed above involve relatively formal, non-immediate use of eye-contact interaction devices, various other modes of use (including less structured, less formal modes) are also possible and useful, for example, for interactive sessions with expected participants and that begin and end at expected times.

[0221] For example, in some cases, it may be advantageous to keep the eye-contact video interactive device constantly or semi-permanently powered and operated. In this method, the interactive device will serve as an open virtual window or entrance on the participant's desk or in other rooms (such as lounges) or in a variety of other public or private locations.

[0222] In effect, the open window or entrance provides remote participants with an informal access channel to engage in electronic activities similar to a "casual visit" or "casual tour" in real life. Remote participants can initiate an interactive session with the local participant at any time and under any circumstances (including unannounced ones) simply by connecting to an always-on eye-contact interactive device and starting a conversation.

[0223] When the eye-contact interactive device is operated in "always-on" mode, it can provide a virtual window or entrance next to the main participant, which others can "casually visit" or "casually access" to, thus approximating the work experience of an "open office," where such casual visitors and chance encounters are more common.

[0224] The increasing habit of working from home reduces spontaneous and frequent face-to-face interactions that occur in the traditional physical workplace. Working from home reduces opportunities for spontaneous conversations and real-time problem-solving with others. Eye contact interaction devices can create virtual environments similar to real-life open office environments, where remote participants can virtually "tap" a local eye contact interaction device and be placed into that virtual environment by a local participant.

[0225] As mentioned earlier, in some instances, a local participant can maintain two or more eye-contact interaction devices to facilitate multiple simultaneous but separate conversations with different remote participants. In some cases, a participant can simultaneously “project” onto more than one eye-contact video interaction device.

[0226] Participant separation

[0227] Eye contact video interaction can occur across a wide variety of physical separation distances. For example, the physical distance between participants does not need to be as large as the separation distances typically implied by different buildings, streets, cities, or countries in a video conference. In some cases, participants can be as close to each other as if they were on opposite sides of a wall, or at two locations in a room, office, shop, or other location, or on two floors of a building.

[0228] More generally, one or more eye-contact interaction devices can be useful in any situation where any physical separation between participants is desired, necessary, or existing. Among other things, situations can include security situations where participants are close to each other (e.g., in a single room or space) but separated by walls, barriers, or screens, and medical situations where participants need to be isolated from each other or otherwise isolated (e.g., to prevent the spread of disease from one to another). The use of eye-contact interaction devices is possible when one participant has limited communication or mobility, such as a patient, and the other participant is a caregiver who may be nearby (in the same building) or at a greater distance. Retail and other business activities can also benefit from the use of eye-contact video interaction when close proximity or contact is risky, undesirable, or impractical. Some applications can include systems for banks, on-site bank tellers, service station managers, building entrances, and a wide variety of other applications. Among other things, any kind of customer service or instructional interaction can benefit from the use of eye-contact interaction devices, especially in situations where participants need to be separated for health, safety, security, personal comfort, or other reasons.

[0229] In some situations, social distancing requirements associated with Covid-19 and other diseases can be met by using eye-contact video interaction devices.

[0230] Application scenarios

[0231] Much of the previous discussion has described the use of local participants in private settings, such as homes or offices. In some implementations, eye-contact video interaction devices can be located temporarily or permanently at various public or semi-public locations.

[0232] For example, in the emergency room of a medical and health center, a patient, as a participant, can be triaged without risk to healthcare workers until it is determined that the patient can be given physical access to the facility.

[0233] In some cases, patients who cannot or are not allowed to have in-person guests (such as those who are in isolation) can have eye contact video interactions with family members or friends.

[0234] For telemedicine appointments or general remote health consultations, eye-to-eye contact via eye-contact interaction devices can improve clinician-patient rapport. Certain clinical examinations (“Let me see your throat” or “Say ‘ah’) can also benefit from a better viewing plane than might be possible with eye-contact interaction devices.

[0235] Job interviews will be a useful application of technology. When international and regional travel is restricted, as has happened, for example during the COVID-19 pandemic, in-person interaction is sometimes impossible in job interviews, and eye contact video interaction can be useful in the process.

[0236] Journalists or podcasters conducting “live” interviews with people who are not physically present can benefit from the high level of engagement offered by eye contact video interaction.

[0237] Other places where eye contact video interaction can be particularly useful include bank teller windows, ATMs, help kiosks located inside or outside buildings or rooms (e.g., in car rental offices, airports, food stores or grocery stores), drive-thru restaurants, hotels (including concierges), security posts, customer support and help desk centers, schools, other educational institutions and prisons, to name just a few.

[0238] Handheld eye-contact video interaction device

[0239] Handheld or portable stand-alone eye-contact interaction devices can be useful in certain situations. Such devices can have one of the previously discussed configurations (including an integrated computing device) and are fully portable by having an internal power source (e.g., a battery), a 5G SIM card (for wireless communication over a cellular network), and a "pistol grip" stand, allowing for comfortable carrying and operation. The bandwidth provided by 5G networks enables mobile, high-resolution, two-way, high-definition video calls with eye-contact video interaction. In some examples, handheld devices can be used in "live television breaking news" interviews where a reporter or bystander can interact with a presenter in the studio through eye-contact video interaction with someone on set.

[0240] Components and features

[0241] In some embodiments, additional sensors or devices or other technologies and combinations thereof may be included in or set as accessories to the eye contact interaction device.

[0242] For example, information related to a local participant can be acquired at the eye-contact interaction device and transmitted to a remote participant for use at another location. In some cases, near-infrared sensors can support facial recognition of the local participant to be used by the remote participant. Such infrared sensors can also be used to measure the temperature of the local participant. In some applications, the remote participant can use the temperature information to, for example, decide whether to grant the local participant access to a building, room, or other space. More generally, for medical or security purposes, eye-contact interaction devices can facilitate the capture and verification of biometric information related to the local participant.

[0243] Types of interaction

[0244] Many of the descriptions above address the use of eye-contact interactive devices in contexts such as meetings, video calls, conferences, webinars, medical appointments, parties, social gatherings, courses, podcasts, lectures, seminars, board meetings, and discussion groups. Typically, such events have start and end times, are pre-arranged, involve identified participants, and are conducted as a gathering to cover one or more topics through participant discussions.

[0245] Many uses and applications are possible in which eye-contact video interaction between local and remote participants is necessary, desired, or preferred.

[0246] In some implementations, local participants can control the position and movement of eye-contact interactive devices in a remote space, enabling them to realistically participate in a remote environment while engaging in activities such as video conferencing. For example, through methods such as... https: / / suitabletech.com The appropriate technology shown in Beam TM Products are designed to provide this type of sport.

[0247] Two-way eye contact interaction

[0248] While some of the examples discussed above involve using local eye-contact interaction devices at the far end of the communication channel compared to traditional non-eye-to-eye devices, the experience is better for both local and remote participants if eye-contact interaction devices are used at both ends.

[0249] Integrated light source

[0250] In some embodiments, an integrated light source may be included within the housing to illuminate the camera's capture area and optimize video quality. The light source may be an LED light mounted on the housing and adjustable in orientation and position. In some implementations, the LED light may be extendable and attached via, for example, miniature ball head screws used on tripods. https: / / www.amazon.com.au / AKOAK-Swivel-Tripod-Camcorder-Bracket / dp / B01CQAQOSI This allows the light to be angled appropriately. See, for example... https: / / www.leedervillecameras.com.au / products / tripods-flashes / led-video-lights / litufoto-f12-led-video-light .

[0251] voice commands

[0252] In some cases where voice commands or sounds are used to control electronic devices (such as Facebook's portal or Google's home hub), incorporating some or all components of an eye-contact interaction device into such devices to provide them with eye-contact interaction capabilities can significantly improve the user experience. When operating an eye-contact interaction device via voice commands, the microphone can be configured to listen for specific voice commands and perform any previously described functions based on the voice. Participants can download and select from any number of third-party voice-driven AI assistants (such as Siri (Apple) or Alexa (Amazon)).

[0253] In some cases, eye-contact interaction devices can track speech as it moves around the room. For example, when this setting is activated, the eye-contact interaction device will pan from left to right to follow a participant who is speaking and moving around the room. The eye-contact interaction device can also amplify the speech of the speaking participant.

[0254] Refractive correction

[0255] In some implementations, the eye-contact video interaction device may include mechanisms for providing refractive correction of the displayed video to accommodate both myopic and hyperopic participants. The refractive correction mechanism may include electronic or manual controls for adjusting the focus of the participant's image.

[0256] Incorporating this technology into various other electronic devices can make them more efficient, useful, and safe.

[0257] Other implementations are also within the scope of the appended claims.

[0258] For example, such as Figure 16 , Figure 17 and Figure 18 As shown, in some implementations, the eye-contact interaction device can take the form of an elegant desktop appliance 600 that also functions as a table lamp.

[0259] The appliance includes a base 602 configured to be heavy enough or otherwise stable enough to support the appliance on a surface without excessive risk of it falling. A stand 604 extends upward from the base to a point 605 where an interaction component 606 is mounted. The stand may be integrated with the base as a continuous element or may be attached to the base separately. The stand may have a top section 608 extending toward the user of the appliance, and the interaction component may be attached to or suspended from a fitting 610 at the upper end of the stand. The stand may accommodate (among other things) cables extending between the base and the interaction component to carry power and signals for various purposes as discussed elsewhere.

[0260] The interactive components include an upper component 612 and a lower component 614 connected in a flip-top manner by a hinge 616. The hinge is configured to achieve two or more stable orientations of the lower component relative to the upper component. In one stable orientation, the lower component is held horizontally, which causes it to face the upper component at a 45-degree angle. In some examples, the upper component is held fixed in an orientation at a 45-degree angle relative to the horizontal plane on which the appliance rests.

[0261] The lower component can be pushed upward from a first stable orientation to cause it to pivot about a hinge and eventually reach a second stable orientation in which it is closed against the upper component. In use, the interactive component will typically have the lower and upper components in the first stable orientation. When the appliance is not in use, the interactive component can remain in the first stable orientation or can be placed in the second stable orientation, where the operating surface of the interactive component is hidden and not visible. In the second orientation, the appliance can be used as... Figure 18 The elegant table lamp shown.

[0262] Magnets, friction, or other mechanisms, or combinations thereof, can be used to keep the interactive component open or closed, or in an intermediate orientation between fully open and closed. When the interactive component is closed, the screen of the video display and the surface of the beam splitter glass are protected.

[0263] The upper and lower components of the interactive component together provide many elements and features of one or more of the above configurations in a compact and elegant component.

[0264] The upper assembly includes a beamsplitter glass 608 that reflects the content displayed on the lower video display 619 and allows light to pass through to a compact camera 610 mounted behind the beamsplitter glass. In some applications, the camera's video capture axis 609 is pointed towards the participant's eyes, and the stand positions the interactive components horizontally such that the video capture axis 609 can be horizontal when pointed towards the participant's eyes. The compact arrangement of the camera is made possible by positioning the camera's circuitry in the lower assembly; the remaining camera sensor and lens are then compact and can be housed behind the beamsplitter glass in the upper assembly. The upper assembly includes a housing 615 that shields the camera's lenses from ambient light to maintain the quality of the captured video. On the other hand, the space between the beamsplitter glass and the video display needs to be shielded to achieve a streamlined design.

[0265] The housing 615 may also include other elements useful in the function of the appliance, including elements described for other configurations of the eye contact interaction device.

[0266] The bottom component includes a video display 619 held within the housing 620, with the display screen facing the beam splitter glass. The housing may also contain one or more microphones 621, 623, and various other elements useful in the function of the appliance, including those described for other configurations of eye-contact interaction devices.

[0267] The housing may also include lamps 622, for example, in the form of horizontal fluorescent lamps or horizontal LED arrays. The emitted light (including its intensity and spectral characteristics) can be turned on and off and otherwise controlled manually from one or more physical switches or wirelessly from a mobile device, or electronically via a computing device that is part of the appliance.

[0268] The lamp can be used to illuminate the face of a participant engaging in eye contact interaction or to illuminate other parts of the camera lens's field of view, or it can function as a desk lamp, for example, when the interactive component is in a secondary orientation, or both. One or more other lamps can be mounted on the interactive component in various positions and orientations. Because of the lamp's dual function, the appliance does not occupy non-functional desk space when it is not used for eye contact video interaction.

[0269] The base includes a housing 630 containing one or more speakers 632, 633; ​​control buttons, including volume control buttons 691, 693; and signal ports, such as an audio jack 634; a small computing device, cable connection points for HDMI 695, USB-C 696 and DC power 697, and various other elements useful in the function of the appliance (including elements described for other configurations of eye-contact interaction devices).

[0270] Housings 615 and 620 can be considered together as housings, a housing that is simpler and possibly more elegant than the housings previously described.

[0271] Hinge 616 is configured to allow for a range of possible stable orientations of the lower component relative to the participant's face, aligning the video capture axis with the participant's gaze axis. Accessory 610 can also be arranged to allow one or more rotational degrees of freedom, enabling the entire interactive component to be oriented as needed. This feature can be used not only to guide the video capture axis when the interactive component is open and during eye-contact video interaction, but also to adjust the direction of light emitted from the lamp when the interactive component is closed.

[0272] Electronic components used for operating the appliance may be located in interactive components, a stand or base, or a combination thereof.

[0273] In some implementations, the eye-contact video interaction device includes a physical mute switch, which can be used in addition to the software mute switch presented by the user interface of the video interaction platform, or the physical mute switch can be used to replace the software mute switch presented by the user interface of the video interaction platform.

[0274] In some implementations, the computing device can be located outside the main structure of the appliance and can be placed on a shelf, for example, as part of the base.

Claims

1. A video device, comprising: The above components include: A beam splitter having a front surface and a rear surface, and A video sensor and an optical lens are mounted on the rear surface of the beam splitter, the video sensor and the optical lens having an optical axis extending from the video sensor through the beam splitter. The following components include: A video display, located at a position facing the front surface of the beam splitter, and A mechanism associated with the beam splitter and the video display is configured to implement an open configuration in which reflections from the screen of the video display are visible from a position facing the front surface of the beam splitter, and a closed configuration in which reflections from the video display are not visible from the position facing the front surface of the beam splitter. A support structure is provided for supporting the beam splitter, the video sensor, the optical lens, the video display, and the mechanism at a height above the working surface, and for maintaining the upper components in a fixed orientation. LEDs are mounted to the lower assembly and extend along the bottom surface of the lower assembly.

2. The video device according to claim 1, wherein, The mechanism includes a hinge coupled between the beam splitter and the video display and configured to allow relative movement between the beam splitter and the video display in the open and closed configurations.

3. The video device according to claim 1, wherein, The LED is a light used to provide illumination in the vicinity of the device.

4. The video device according to claim 3, wherein, The light is exposed in both the open configuration and the closed configuration.

5. The video device according to claim 1, wherein, The beam splitter is configured to align the video capture axis of the video sensor with the display axis of the video display along a common axis, such that when the video sensor is capturing at least the eyes of a first person looking toward the video sensor along the common axis, the first person is able to view at least the eyes of a second person along the common axis in real time.

6. The video device according to claim 5, wherein, The support structure includes a computing device configured to maintain a video interaction session between the first and second persons using the output of the video sensor and the input of the video display.

7. The video device according to claim 6, wherein, The computing device includes a minicomputer.

8. The video device according to claim 6, wherein, The eye contact video interaction session includes video conferencing.

9. The video device according to claim 6, wherein, The aforementioned eye-contact video interaction session is conducted between a first person and a second person within the same building or space.

10. The video device according to claim 6, wherein, The aforementioned eye-contact video interaction session is conducted between a first person and a second person who are not in the same building or the same space.

11. The video device of claim 6, further comprising a second computing device, the second computing device being configured to execute instructions to provide functions other than maintaining the eye contact video interaction session.

12. The video device of claim 6, further comprising a wireless mouse and keyboard configured to control the computing device.

13. The video device according to claim 1, wherein, The support structure includes a base, the base including electronic components, the electronic components including one or more of a microphone, a speaker, and a computing device.

14. The video device according to claim 6, wherein, The video display is configured to present a user interface so that the first person can control the eye contact video interaction session through the user interface.

15. A method for operating a video device according to any one of claims 1-14, the method comprising: The video sensor captures at least the first eye view of the first person watching along the video capture axis. A second video, at least the eyes of a second person, is presented along the display axis via the video display. The beam splitter aligns the video capture axis and the display axis along a common axis, such that while the first video of the first person's eyes is being captured along the common axis, the first person can view at least the second video of the second person's eyes along the common axis. The instructions are executed to maintain a video interaction session of eye contact between the first and second persons using the captured and presented video.

Images