Vergence-Based Gaze Matching for Mixed-Mode Immersive Telepresence Applications

Abstract

Aspects of the present disclosure provide a method and apparatus for gaze matching. In some embodiments, a processing circuit determines a position of an object of interest of a first user and receives a first one or more images of the first user captured by a camera at a camera position different from the position of the object of interest. The processing circuit detects a first convergence of eyes of the first user, calculates a mismatch of the first convergence to view the object of interest, and performs gaze correction of the first one or more images based on the mismatch of the first convergence to view the object of interest.

Description

[Technical field]

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 359,746, entitled "Convergence-Based Gaze Matching for Mixed-Mode Immersive Telepresence Applications," filed on July 8, 2022, which claims the benefit of priority to U.S. Provisional Application No. 18 / 207,594, entitled "Convergence-Based Gaze Matching for Mixed-Mode Immersive Telepresence Applications," filed on June 8, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments generally related to media processing, including real-time immersive telepresence applications. [Background technology]

[0003] The "Background" discussion provided herein is intended to generally present the context of the present disclosure. To the extent that it is described in this Background section, the work of the current inventors, as well as aspects of the disclosure that may not otherwise be admitted as prior art at the time of filing, are not expressly or impliedly admitted as prior art to the present disclosure.

[0004] Real-time immersive telepresence applications, such as video chat, training, education, and the like, allow people in remote locations to have real-time conversations, training, and various pedagogical teaching models. In some embodiments, during a real-time immersive telepresence application, a display screen is placed in front of a first user, and the display screen can display an image of a second user to simulate a face-to-face environment. Summary of the Invention [Problem to be solved by the invention]

[0005] Aspects of the present disclosure provide methods and apparatus for gaze matching. [Means for solving the problem]

[0006] In some embodiments, a processing circuit determines a position of an object of interest of a first user, receives a first one or more images of the first user captured by a camera at a camera position different from the position of the object of interest, detects a first vergence or rotation of eyes of the first user, calculates a mismatch of the first vergence or rotation to view the object of interest, and performs gaze correction of the first one or more images based on the mismatch of the first vergence or rotation to view the object of interest.

[0007] In some embodiments, the processing circuitry receives a second one or more images of a second user, the second user and the first user engaging in immersive telepresence, the processing circuitry derives an interpupillary position of the second user from the second one or more images, displays the second one or more images with the interpupillary position set in a screen plane of a display screen of the first user, and determines the interpupillary position in the screen plane of the display screen as the position of the object of interest of the first user.

[0008] In some embodiments, the processing circuitry determines a centre point of the first user's display screen as the location of the first user's object of interest.

[0009] In some embodiments, the processing circuitry receives a second one or more images of a second user engaging in immersive telepresence with the first user, displays the second one or more images on a display screen of the first user, and determines a position on the display screen for displaying the eyes of the second user as the position of the object of interest of the first user.

[0010] In one example, the first vergence or rotation is sensed based on an eye tracking sensor separate from the camera, and in another example, the first vergence or rotation is detected based on image analysis of the first one or more images of the first user.

[0011] In some embodiments, the processing circuitry calculates the first convergence or rotation of the eyes of the first user based on a head position of the first user.

[0012] In some embodiments, the processing circuitry determines a point of gaze according to the first convergence or rotation of the eyes of the first user and a head position of the first user, and calculates the mismatch of the point of gaze relative to the position of the object of interest of the first user.

[0013] In one example, the processing circuitry modifies a head position of the first user in the first one or more images. In another example, the processing circuitry modifies a body position of the first user in the first one or more images. In another example, the processing circuitry modifies a head pose of the first user in the first one or more images. In another example, the processing circuitry modifies a body pose of the first user in the first one or more images. In another example, the processing circuitry modifies a head rotation angle of the first user in the first one or more images. In another example, the processing circuitry modifies a body rotation angle of the first user in the first one or more images. In another example, the processing circuitry modifies an eye convergence angle or rotation angle of the first user in one or more images.

[0014] In some embodiments, the processing circuitry determines the location of the object of interest according to at least one of a facial expression of a person on a display screen, a visual sentiment analysis of the person on the display screen, and a mood of the person on the display screen.

[0015] In some embodiments, the processing circuitry detects the first convergence or rotation of the eyes of the first user using at least one of an inertial measurement unit (IMU), a depth sensor, a light detection and ranging (LiDar) sensor, a near infrared (NIR) sensor, and a spatial audio detector.

[0016] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the method of gaze matching. [Brief description of the drawings]

[0017] [Figure 1] 1 shows an illustration of gaze correction according to some embodiments of the present disclosure. [Diagram 2] FIG. 1 illustrates a diagram of an immersive telepresence system in some embodiments. [Diagram 3] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0019] In various scenarios, it may not be possible to position the camera directly in front of a person. Thus, the picture of the person captured by the camera may show a different gaze than the real gaze of the real person. According to one aspect of the present disclosure, in immersive telepresence applications, gaze connection between the real person and the image of the person can improve immersion, enhance perception, and in some examples, can be a ubiquitous cue.

[0020] Some aspects of the present disclosure provide techniques for convergence or rotation based image gaze adjustment, such as gaze correction and gaze alignment, which can compensate for camera position and match user gaze in immersive telepresence applications.

[0021] According to some aspects of the present disclosure, the convergence-based image gaze adjustment can detect the convergence or rotation of the user's eyes and determine whether the convergence, the user's eye position, and the head position are consistent with the user viewing an object of interest. For example, gaze can refer to fixed visual attention by the user, and the user's gaze position (also called gaze position) in the image can be determined based on the user's eye convergence or rotation, the user's eye position, and the head pose in the image. Furthermore, gaze correction parameters can be determined to match the gaze position with the position of the object of interest. For example, head rotation parameters and eye rotation parameters can be determined to adjust the gaze position to match the position of the object of interest. In some examples, one or more images of the user can be taken, and the head position and eye position in the one or more images can be adjusted according to the gaze correction parameters. Thus, the user in the one or more images can have, for example, an appropriate gaze on the object of interest.

[0022] Figure 1 shows an illustration of gaze correction according to some embodiments of the present disclosure. In the example of Figure 1, a user (101) (e.g., a person) is in front of a display screen (120). The display screen (120) can be part of an electronic system (110), such as a computer system, an entertainment system, a gaming system, a communication system, an immersive telepresence system, etc. The components of the electronic system (110) can be separate components connected by wired or wireless connections, or can be integrated into a package.

[0023] The electronic system (110) includes a display screen (120) and a camera (130) and other suitable components, such as one or more processors (not shown, also referred to as processing circuitry), communication components (not shown), etc. In the example of Figure 1, the camera (130) is located on top of the display screen (120). However, the camera (130) can be located in other suitable locations, such as on the side of the display screen (120).

[0024] Typically, the user (101) is in front of and faces the display screen (120), which is at about face height for the user (101) to comfortably view the content displayed on the display screen (120). The camera (130) is typically positioned around the periphery of the display screen (120) and is not positioned directly in front of the user (101) so as not to interfere with the user's (101) viewing of the content displayed on the display screen (120).

[0025] According to one aspect of the present disclosure, an image of the user (101) captured by the camera (130) while the user (101) is looking at the display screen (120) may indicate that the user (101) is gazing in a direction other than approximately the center of the display screen (120). For example, while the user (101) is looking at a central position of the display screen (120), an image of the user (101) captured by the camera (130) may indicate that the user (101) is looking at a position lower than the central position of the display screen (120).

[0026] In another example, a user (101) is a first user communicating with a second user via a telepresence application. The user (101) may see a note posted to the left of the display screen (120) as the camera (130) captures an image of the user (101). When the image is transmitted and displayed on a second display screen in front of the second user, the image of the user (101) on the second display screen does not appear to be looking in the direction of the second user, so the second user does not perceive eye contact with the first user.

[0027] In the example of FIG. 1, gaze correction based on vergence or rotation can be performed, for example, by the electronic system (110) to compensate for camera position and / or to match the gaze of a second user for eye contact. For example, the electronic system (110) can detect eye convergence of the user (101). Eye convergence is the movement of both eyes in opposite directions (also called inward rotation) to obtain or maintain single binocular vision. Eye convergence is then used to perform gaze adjustment. It should be noted that other suitable rotational eye movements can be used for gaze correction, such as any combination of vertical rotational eye movements, horizontal eye movements, or rotational eye movements. In some embodiments, the gaze position (102) of the user (101) can be determined by eye convergence and other suitable information, such as the head position and / or eye position of the user (101). The gaze position (102) is compared to the position (111) of the object of interest of the user (101) to determine gaze correction parameters. For example, gaze correction parameters can be applied to the user (101) to adjust the corrected gaze position to position (111). In some embodiments, the gaze correction parameters can be used to process an image of the user (101) such that the user (101) in the processed image appears to be gazing at an object of interest.

[0028] It should be noted that eye convergence can be detected by a variety of techniques. In one example, a physical eye tracking sensor is used to detect eye convergence. In another example, image analysis can be performed on an image of the user (101) to detect eye convergence. In another example, the user's head position alone is used to detect eye convergence of the user (101).

[0029] In some implementations, head position and eye convergence are used to perform gaze adjustments that can compensate for camera position and match the gaze of a second user.

[0030] It should be noted that the location of the object of interest can be determined by various techniques. In one example, a center point of the display screen (120) is determined to be the object of interest. In another example, an image of a second user is displayed on the display screen (120) and an eye position of the second user on the display screen (120) is determined to be the location of the object of interest. In another example, the location of the object of interest is determined according to at least one of a facial expression of the person on the display screen, a visual emotion analysis of the person on the display screen, and a mood of the person on the display screen.

[0031] In some embodiments, the image of the user (101) captured by the camera (130) is modified. For example, the position, pose, or rotation of the body of the user (101) in the image may be adjusted. In other embodiments, the position, pose, or rotation of the head of the user (101) in the image may be adjusted. In other embodiments, the position and rotation of the eyes of the user (101) in the image may be adjusted.

[0032] It should be noted that while the components of FIG. 1 are shown for illustrative purposes, a variety of different components can be used to implement the electronic system (110). For example, the electronic system (110) is configured for immersive telepresence. For example, the display screen (120) can be implemented with a 3D display, such as an 8K autostereoscopic display with left and right view cones. The camera (130) can be implemented with a high speed RGB camera, a stereo camera, a multi-camera, a depth capture camera, and the like. In one example, the camera (130) is implemented with a 4K stereoscopic RGB camera. The electronic system (110) can include a high speed transceiver, such as a transceiver capable of live 8K3D transmission. In some implementations, the electronic system (110) can include an eye tracking sensor separate from the camera (130).

[0033] According to some aspects of the present disclosure, due to convergence-based gaze alignment, convergence-based gaze adjustments can be performed in immersive telepresence applications to improve the immersive experience.

[0034] FIG. 2 illustrates a diagram of an immersive telepresence system (200) in some embodiments. The immersive telepresence system (200) includes a first electronic system (210A) and a second electronic system (210B) connected via a network (205). The immersive telepresence system (200) can perform two-way real-time gaze matching in some embodiments. In some embodiments, the immersive telepresence application is hosted by a server device, and the first electronic system (210A) and the second electronic system (210B) can be client devices of the immersive telepresence application.

[0035] According to some aspects of the present disclosure, a virtual background of an immersive telepresence application can be rendered on a client, such as a first electronic system (210A) and a second electronic system (210B), by a virtual camera, and a dynamic human subject in the foreground can be captured with a stereoscopic camera array. In some embodiments, vergence refers to the inward / outward rotation of the eyes to fixate an object, and accommodation refers to the focusing mechanism of the eyes to produce a sharp image on the retina. To avoid the inconsistency of vergence and accommodation by a user, a horizontal image translation technique can be used to set the depth position of the user's object of interest (e.g., the person's eyes on the display screen) to the screen plane of the display screen, which is also called the zero parallax position, which can match the user's focus, accommodation, and vergence in some embodiments. The inward vergence rotation of the user's eyes can be calculated when the user is away from the camera, and gaze correction can be applied when the user is actually looking at the person's eyes on the display screen.

[0036] The first electronic system (210A) and the second electronic system (210B) may each be configured similarly to the electronic system (110).

[0037] For example, the first electronic system (210A) includes a display screen (220A) implemented as a 3D display such as an 8K autostereoscopic display with left and right view cones or any number of view cones. The first electronic system (210A) includes a camera (230A) implemented as a high speed RGB camera, stereo camera, multi-camera, depth capture camera, etc. The first electronic system (210A) includes a transceiver (not shown, wired or wireless) configured to transmit signals to and / or receive signals from the network (205). In some embodiments, the first electronic system (210A) can include an eye tracking sensor (not shown) separate from the camera (230A). The first electronic system (210A) also includes processing circuitry such as one or more processors (not shown) for image processing.

[0038] Similarly, the second electronic system (210B) includes a display screen (220B) implemented as a 3D display such as an 8K autostereoscopic display with left and right view cones or any number of view cones. The second electronic system (210B) includes a camera (230B) implemented as a high speed RGB camera, stereo camera, multi-camera, depth capture camera, etc. The second electronic system (210B) includes a transceiver (not shown, wired or wireless) configured to transmit signals to and / or receive signals from the network (205). In some embodiments, the second electronic system (210B) can include an eye tracking sensor separate from the camera (230B). The second electronic system (210B) also includes processing circuitry such as one or more processors (not shown) for image processing.

[0039] The immersive telepresence system (200) can perform two-way real-time gaze matching. In the example of FIG. 2, the first user (201A) is in front of the display screen (220A) and faces the display screen (220A), and the display screen (220A) is at about the height of the first user's face so that the first user (201A) can comfortably view the content displayed on the display screen (220A). The camera (230A) is disposed around the display screen (220A) and is not disposed directly in front of the first user (201A) so as not to obstruct the first user (201A) from viewing the content displayed on the display screen (220A). For example, the camera (230A) is disposed on the left side of the display screen (220A) in FIG. 2.

[0040] Similarly, the second user (201B) is in front of the display screen (220B) and faces the display screen (220B), and the display screen (220B) is at about the height of the face of the second user (201B) so that the second user (201B) can comfortably view the content displayed on the display screen (220B). The camera (230B) is disposed around the display screen (220B) and is not disposed directly in front of the second user (201B) so as not to obstruct the second user (201B) from viewing the content displayed on the display screen (220B). For example, the camera (230B) is disposed above the display screen (220B) in FIG. 2.

[0041] According to some aspects of the present disclosure, the camera (230A) captures a first stereo image of the first user (201A). The first stereo image can be processed, for example, according to convergence-based gaze matching. In one example, the first stereo image is processed by a processing circuit of the first electronic system (210A) according to convergence-based gaze matching. In another example, the first stereo image is processed by a server (e.g., an immersive telepresence server) in the network (205) according to convergence-based gaze matching. The processed first stereo image is transmitted to the second electronic system (210B). The processed first stereo image is displayed by the display screen (220B) to show a modified image, such as the display image (202B) of the first user (201A) in FIG. 2. In some embodiments, the first stereoscopic image or the processed first stereoscopic image can be further processed by processing circuitry of the second electronic system (210B) for gaze alignment, and the processed first stereoscopic image is then displayed by the display screen (220B) to show a modified image, such as the display image (202B) of the first user (201A).

[0042] Similarly, the camera (230B) captures a second stereo image of the second user (201B). The second stereo image can be processed, for example, according to convergence-based gaze matching. In one example, the second stereo image is processed by a processing circuit of the second electronic system (210B) according to convergence-based gaze matching. In another example, the second stereo image is processed by a server (e.g., an immersive telepresence server) in the network (205) according to convergence-based gaze matching. The processed second stereo image is transmitted to the first electronic system (210A). The processed second stereo image is displayed by the display screen (220A) to show a modified image, such as the display image (202A) of the second user (201B) in FIG. 2. In some embodiments, the second stereoscopic image or the processed second stereoscopic image can be further processed by processing circuitry of the first electronic system (210A), and then the further processed second stereoscopic image is displayed by the display screen (220A) to show a modified image, such as the displayed image (202A) of the second user (201B).

[0043] In some embodiments, when the first user (201A) looks at the displayed image (202A), the first user (201A) makes eye contact (also called gaze alignment) with the displayed image (202A) of the second user (201B). The eye contact enhances the immersive experience of the first user (201A).

[0044] Similarly, when the second user (201B) looks at the display image (202B), the second user (201B) makes eye contact (also called gaze alignment) with the display image (202B) of the first user (201A). The eye contact enhances the immersive experience of the second user (201B). Note that in some embodiments, the first user (201A) and the second user (201B) do not need to simultaneously look at the display image and make eye contact.

[0045] In some embodiments, the first electronic system (210A) receives a processed second stereoscopic image of the second user (201B). In some embodiments, the processed second stereoscopic image includes a pair of images. For each image of the pair of images, determine an interpupillary position (e.g., a center position of two pupils) of the person (e.g., the second user (201B)) in the image. For example, determine two pupils in the image and determine 3D XYZ coordinates of the interpupillary positions of the two pupils. Using the interpupillary positions of the pair of images, the pair of images are shifted and displayed on the display screen (220A) such that the interpupillary positions overlap and are displayed at one position on the display screen (220A), thereby setting the eye depth plane to the screen plane of the display screen (220A). The person's eyes displayed according to the shifted images can then be observed on the screen plane of the display screen (220A), and the screen plane of the display screen (220A) can be referred to as the zero parallax position (ZPS). In some embodiments, when the eyes are observed at the screen plane, the nose can be observed in front of the screen plane and the back of the head is behind the screen plane. In one example, the location of overlapping interpupillary positions on the display screen (220A), as shown at (203A), is defined as the object of interest of the first user (201A) to achieve gaze alignment.

[0046] Further, in the example of FIG. 2, the eye convergence of the first user (201A) is detected. In one example, the eye tracking sensor can determine the eye movement of the first user (201A). The eye movement can be used to determine a 3D vector of the convergence of the eyes. In another example, the 3D vector of the convergence of the eyes is determined according to an image analysis of the first stereoscopic image of the first user (201A). In some embodiments, the head rotation (or head position, or head pose) can be determined from the image analysis of the first stereoscopic image of the first user (201A).

[0047] Eye convergence is then used to perform gaze adjustment. In some embodiments, the gaze position of the first user (201A) can be determined by the eye convergence of the first user (201A) and other suitable information such as head rotation and eye position. The gaze position of the first user (201A) is compared with the position of the object of interest of the first user (201A) to determine gaze correction parameters for the first stereoscopic image. The first stereoscopic image can be processed according to the gaze correction parameters.

[0048] In some embodiments, the first electronic system (210A) includes a sensing module configured to detect eye convergence of the first user (201A). In one example, the sensing module can be realized by a camera (230A), such as a high-speed RGB camera, a stereo camera, a multi-camera, a depth capture camera, etc. In another example, the sensing module can be realized by an eye tracking sensor.

[0049] In some embodiments, the first electronic system (210A) includes a rendering module configured to set a depth plane of a person's eyes in the processed second stereoscopic image relative to a screen plane of the display screen (220A). For example, the rendering module may set a depth plane of a person's eyes in the processed second stereoscopic image relative to a screen plane of the display screen (220A). In some embodiments, the rendering module may be implemented by one or more processors executing software instructions.

[0050] In some embodiments, the first electronic system (210A) includes a registration module configured to determine a disparity between a gaze point and an object of interest. The gaze point can be determined based on eye convergence detected by eye tracking and other suitable information. In some embodiments, the registration module can include a compact model and can be implemented by one or more processors executing software instructions.

[0051] In some embodiments, the first electronic system (210A) includes a gaze correction module configured to perform gaze correction in the first stereoscopic image. In some embodiments, the gaze correction module can receive live input from the sensing module and the registration module and perform corrections to the first stereoscopic image. For example, the gaze correction module can perform 2D restoration and / or 3D rotation gaze correction using the live input from the sensing module and the registration module. In one example, the person in the first stereoscopic image is in the form of a 3D mesh. The 3D mesh can be rotated for gaze correction. In one example, the person's eyes can be rotated for gaze correction. In another example, the person's head can be rotated for gaze correction. In another example, the person's body can be rotated for gaze correction. In some embodiments, the gaze correction module is realized by a neural network model that is trained to serve that purpose.

[0052] In one example, due to the location of the camera (230A), when the first user (201A) looks at the display image (202A), for example, the center of the display screen (220A), the first stereoscopic image captured by the camera (230A) shows that the first user (201A) looks at the right part of the display screen (220A). The first electronic system (210A) performs a gaze match based on convergence to determine gaze correction. For example, the gaze point of the first user (201A) can be determined based on eye convergence, and the gaze point can be a point on the right part of the display screen (220A). A difference between the gaze point and the position of the object of interest (for example, the interpupillary position in the screen plane of the display screen (220A)) can be determined. Based on this difference, a rotation angle for rotating the person in the first stereoscopic image to the left can be determined to correct the first stereoscopic image. Then, a mesh representing the person in the first stereoscopic image can be rotated to the left according to the rotation angle for gaze correction.

[0053] In the above description, the first stereoscopic image is processed by the first electronic system (210A) for gaze correction, however, it should be noted that in some embodiments, gaze correction may also be performed by the second electronic system (210B), provided that in some embodiments, gaze correction may be suitably performed by a server in the network (205).

[0054] Similarly, in some embodiments, the second electronic system (210B) receives the processed first stereoscopic image of the first user (201A). In some embodiments, the processed first stereoscopic image includes a pair of images. For each image of the pair of images, determine the interpupillary position (e.g., the center position of the two pupils) of the person (first user (201A)) in the image. For example, determine the two pupils in the image and determine the 3D XYZ coordinates of the interpupillary positions of the two pupils. Using the interpupillary positions of the pair of images, the pair of images are shifted (also referred to as horizontal image translation in some embodiments) to be displayed on the display screen (220B) such that the interpupillary positions overlap and are displayed at one point, and the eye depth plane is set to the screen plane of the display screen (220B). The person's eyes displayed according to the shifted images can then be observed on the screen plane of the display screen (220B), and the screen plane of the display screen (220B) can be referred to as the zero parallax position (ZPS). In some embodiments, when the eyes are observed at the screen plane, the nose can be observed in front of the screen plane and the back of the head is behind the screen plane. In one example, a point of overlapping interpupillary position on the display screen (220B), as shown at (203B), is defined as the object of interest of the second user (201B) to achieve gaze alignment.

[0055] Further, in the example of FIG. 2, eye convergence of the second user (201B) is detected. In one example, the eye tracking sensor can determine the eye movement of the second user (201B). The eye movement can be used to determine a 3D vector of the convergence of the eyes. In another example, the 3D vector of the convergence of the eyes is determined according to an image analysis of the second stereoscopic image of the second user (201B). In some embodiments, head rotation (or head position, or head pose) is determined from the image analysis of the second stereoscopic image of the second user (201B).

[0056] Eye convergence is then used to perform gaze adjustment. In some embodiments, the gaze position of the second user (201B) can be determined by the eye convergence of the second user (201B) and other suitable information such as head rotation and eye position. The gaze position of the second user (201B) is compared with the position of the object of interest of the second user (201B) to determine gaze correction parameters for the second stereoscopic image. The second stereoscopic image can be processed according to the gaze correction parameters.

[0057] In some embodiments, the second electronic system (210B) includes a sensing module configured to detect eye convergence of the second user (201B). In one example, the sensing module can be realized by a camera (230B), such as a high-speed RGB camera, a stereo camera, a multi-camera, a depth capture camera, etc. In another example, the sensing module can be realized by an eye tracking sensor.

[0058] In some embodiments, the second electronic system (210B) includes a rendering module configured to set a depth plane of a person's eyes in the processed first stereoscopic image relative to a screen plane of the display screen (220B). For example, the rendering module may set a depth plane of a person's eyes in the processed first stereoscopic image relative to a screen plane of the display screen (220B). In some embodiments, the rendering module may be implemented by one or more processors executing software instructions.

[0059] In some embodiments, the second electronic system (210B) includes a registration module configured to determine a disparity between a gaze point and an object of interest. The gaze point can be determined based on eye convergence detected by eye tracking and other suitable information. In some embodiments, the registration module can be implemented by one or more processors executing software instructions.

[0060] In some embodiments, the second electronic system (210B) includes a gaze correction module configured to perform gaze correction in the second stereoscopic image. In some embodiments, the gaze correction module can receive live input from the sensing module and the registration module and perform corrections to the second stereoscopic image. For example, the gaze correction module can perform 2D restoration and / or 3D rotation gaze correction using the live input from the sensing module and the registration module. In one example, the person in the second stereoscopic image is in the form of a 3D mesh. The 3D mesh can be rotated for gaze correction. In one example, the person's eyes can be rotated for gaze correction. In another example, the person's head can be rotated for gaze correction. In another example, the person's body can be rotated for gaze correction. In some embodiments, the gaze correction module is realized by a neural network model that is trained to serve that purpose.

[0061] In one example, due to the location of the camera (230B), when the second user (201B) looks at the display image (202A), for example, the center of the display screen (220B), the second stereoscopic image captured by the camera (230B) shows that the second user (201B) looks at the bottom of the display screen (220B). The second electronic system (210B) performs a gaze match based on convergence to determine gaze correction. For example, the gaze point of the second user (201B) can be determined based on eye convergence, and the gaze point can be a point at the bottom of the display screen (220B). The difference between the gaze point and the position of the object of interest (for example, the interpupillary position in the screen plane of the display screen (220B)) can be determined. Based on this difference, a rotation angle for rotating the person's head or body in the second stereoscopic image upwards can be determined to correct the second stereoscopic image. The mesh representing the person in the second stereo image can then be rotated upwards according to the rotation angle for gaze correction.

[0062] In the above description, the second stereoscopic image is processed by the second electronic system (210B) for gaze correction, however, it should be noted that in some embodiments, gaze correction can also be performed by the first electronic system (210A), provided that in some embodiments, gaze correction can be suitably performed by a server in the network (205).

[0063] 3 shows a flow chart outlining a process (300) according to one embodiment of the present disclosure. In various embodiments, the process (300) can be performed by a processing circuit in an electronic system, such as the electronic system (110), the first electronic system (210A), or the second electronic system (210B). In some embodiments, the process (300) is implemented in software instructions, such that the processing circuit performs the process (300) when the processing circuit executes the software instructions. The process starts at (S301) and proceeds to (S310).

[0064] At (S310), a location of an object of interest of a user, such as a first user, is determined.

[0065] At (S320), a first image or images of a first user captured by a camera at a camera position different from the position of the object of interest are received.

[0066] In (S330), a first convergence or rotation of the first user's eye is determined.

[0067] In (S340), a first vergence or rotation mismatch for viewing the object of interest is determined.

[0068] At (S350), gaze correction is performed on the first image or images based on a first vergence or rotation mismatch for viewing the object of interest.

[0069] In some embodiments, to determine a location of the object of interest, a second one or more images of a second user engaging in immersive telepresence with the first user are received. An interpupillary position of the second user is then derived from the second one or more images. The second one or more images are displayed with the interpupillary position set in a screen plane of a display screen of the first user. The interpupillary position in the screen plane of the display screen is determined as the location of the object of interest of the first user.

[0070] In some embodiments, a second image or images of a second user engaging in immersive telepresence with the first user are received, the second image or images are displayed on a display screen of the first user, and a location on the display screen for displaying the eyes of the second user is determined as a location of an object of interest of the first user.

[0071] In some embodiments, the center point of the display screen is determined as the location of the object of interest of the first user.

[0072] To detect a first convergence of the eyes of the first user, in one example, the first convergence is determined based on data sensed by an eye tracking sensor separate from the camera. In some embodiments, the first convergence is determined based on image analysis of the first one or more images of the first user.

[0073] In one example, a first convergence of the eyes of the first user is calculated based solely on the head position of the first user.

[0074] In some embodiments, a gaze point is determined according to the first convergence of the eyes of the first user and the head position of the first user to further calculate a first convergence disparity for viewing the object of interest. Then, a difference of the gaze point with respect to the position of the object of interest of the first user is calculated to determine the disparity.

[0075] In some embodiments, gaze correction can be performed by adjusting various parameters such as the position of the head and / or body of the first user, the pose of the head and / or body of the first user, and the rotation angle of the head and / or body of the first user. In one example, the head position of the first user in the first one or more images is modified. In another example, the body position of the first user in the first one or more images is modified. In another example, the head pose of the first user in the first one or more images is modified. In another example, the body pose of the first user in the first one or more images is modified. In another example, the head rotation angle of the first user in the first one or more images is modified. In another example, the body rotation angle of the first user in the first one or more images is modified.

[0076] In some embodiments, the location of the object of interest is determined according to at least one of a facial expression of the person on the display screen, a visual emotion analysis of the person on the display screen, and a mood of the person on the display screen.

[0077] In some embodiments, a first convergence of the eyes of the first user is detected using at least one of an inertial measurement unit (IMU), a depth sensor, a light detection and ranging (LiDar) sensor, a near infrared (NIR) sensor, and a spatial audio detector.

[0078] The process then proceeds to (S399) and ends.

[0079] The process (300) may be adapted as appropriate. Steps of the process (300) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0080] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media, such as non-transitory computer-readable storage devices. For example, Figure 4 illustrates a computer system (400) suitable for implementing certain embodiments of the disclosed subject matter.

[0081] Computer software may be encoded using any suitable machine code or computer language, and may be produced by mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or may be executed by interpretation, microcode execution, etc.

[0082] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0083] 4 are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (400).

[0084] The computer system (400) may include certain human interface input devices that may respond to input by one or more users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). The human interface input devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic vision, etc.).

[0085] The input human interface devices may include one or more (only one of which is shown) of a keyboard (401), a mouse (402), a trackpad (403), a touch screen (410), a data glove (not shown), a joystick (405), a microphone (406), a scanner (407), and a camera (408).

[0086] The computer system (400) may also include a number of human interface output devices that may stimulate one or more of the user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (410), data gloves (not shown), or joystick (405), but may also be haptic feedback devices that do not act as input devices), audio output devices (e.g., speakers (409), headphones (not shown)), visual output devices (e.g., screens (410), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may output two dimensional visual output or three or more dimensional output by means of stereographic output, virtual or augmented reality glasses (not shown), multi-view displays, holographic displays and smoke tanks (not shown), and printers (not shown)).

[0087] The computer system (400) may also include human accessible storage devices and their associated media, such as optical media (421), including, for example, CD / DVD ROM / RW (420) with CD / DVD or similar media, thumb drives (422), removable hard drives or solid state drives (423), legacy magnetic media (not shown), such as tape and floppy disks, and dedicated ROM / ASIC / PLD based devices (not shown), such as security dongles.

[0088] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0089] The computer system (400) may also include an interface (454) to one or more communication networks (455). The network may be, for example, wireless, wired, or optical. The network may further be a local network, a wide area network, a metropolitan area network, a vehicular network, an industrial network, a real-time network, a delay-tolerant network, and the like. Examples of networks include Ethernet, WLAN, GSM, cellular networks including 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial TV, vehicular networks including CANBus, and industrial networks, and the like. Some networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (449) (e.g., a USB port of the computer system (400)), while others are typically integrated into the core of the computer system (400) by connecting to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system (400) can communicate with other entities. Such communication may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from the CAN bus to a particular CAN bus device), or bidirectional, e.g., to other computer systems using local or wide area digital networks. As noted above, specific protocols and protocol stacks may be used with each of these networks and network interfaces.

[0090] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to a core (440) of the computer system (400).

[0091] The cores (440) may include one or more central processing units (CPUs) (441), graphics processing units (GPUs) (442), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (443), hardware accelerators for specific tasks (444), graphics adapters (450), etc. These devices may be connected via a system bus (448) along with read-only memory (ROM) (445), random access memory (446), and internal mass storage (447), such as an internal hard disk, SSD, etc. that is not accessible to the user. In some computer systems, the system bus (448) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (448) or via a peripheral bus (449). In some embodiments, a screen (410) may be connected to the graphics adapter (450). Peripheral bus architectures include PCI, USB, etc.

[0092] The CPU (441), GPU (442), FPGA (443), and accelerator (444) may execute a number of instructions, which may combine to constitute the above computer code. The computer code may be stored in a ROM (445) or a RAM (446). Transient data may also be stored in the RAM (446), while permanent data may be stored, for example, in an internal mass storage (447). The use of a cache memory, which may be closely associated with one or more of the CPU (441), GPU (442), mass storage (447), ROM (445), RAM (446), etc., may enable fast storage and retrieval from any memory device.

[0093] The computer-readable media can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0094] By way of example, and not limitation, a computer system having the architecture (400), and in particular the core (440), may provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as specific storage of the core (440) that is non-transitory in nature, such as the core internal mass storage (447) or ROM (445). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (440). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core (440), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to execute the particular processes or particular portions of the particular processes described herein, including the definition of data structures stored in RAM (446) and the modification of such data structures according to the processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (444)), which may operate in place of or in conjunction with software to perform particular processes, or portions of particular processes, described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs), etc.) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0095] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to invent numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. determining a location of an object of interest of a first user; receiving a first one or more images of the first user captured by a camera at a camera position different from the position of the object of interest; detecting a first convergence of an eye of the first user; calculating the first convergence discrepancy for viewing the object of interest; performing gaze correction of the first one or more images based on the inconsistency of the first convergence to view the object of interest; 23. A method for gaze alignment comprising:

2. The step of determining the location of the object of interest comprises: receiving a second one or more images of a second user, the second user and the first user engaging in immersive telepresence; deriving an interpupillary position of the second user from the second one or more images; displaying the second image or images with the interpupillary position set to a screen plane of a display screen of the first user; 2. The method of claim 1, further comprising: determining the interpupillary position in the screen plane of the display screen as the position of the object of interest of the first user.

3. The step of determining the location of the object of interest comprises:

2. The method of gaze matching of claim 1, further comprising the step of determining a center point of the display screen of the first user as the location of the object of interest of the first user, the camera position being at the periphery of the display screen.

4. The step of determining the location of the object of interest comprises: receiving a second one or more images of a second user, the second user and the first user engaging in immersive telepresence; displaying the second image or images on a display screen of the first user, the camera position being at the periphery of the display screen; 2. The method of gaze matching of claim 1, further comprising: determining a position on the display screen for displaying the eyes of the second user as the position of the object of interest of the first user.

5. The step of detecting the first convergence of the eyes of the first user includes: sensing the first convergence based on an eye tracking sensor separate from the camera; and detecting the first convergence based on image analysis of the first one or more images of the first user.

6. The step of detecting the first convergence of the eyes of the first user includes:

2. The method of gaze congruence of claim 1, further comprising: calculating the first convergence of the eyes of the first user based on a head position of the first user.

7. The step of calculating the disparity of the first congestion for viewing the object of interest comprises: determining a gaze point according to the first convergence of the eyes of the first user and a head position of the first user; 2. The method of claim 1 further comprising: calculating the mismatch of the point of gaze relative to the location of the object of interest of the first user.

8. The step of performing gaze correction includes: modifying a head position of the first user in the first one or more images; modifying a body position of the first user in the first one or more images; correcting a head pose of the first user in the first one or more images; modifying a body pose of the first user in the first one or more images; modifying a head rotation angle of the first user in the first one or more images; and modifying a body rotation angle of the first user in the first one or more images.

9. The step of determining the location of the object of interest of the first user further comprises:

2. The method of gaze matching of claim 1, further comprising: determining the location of the object of interest according to at least one of a facial expression of a person on a display screen, a visual emotion analysis of the person on the display screen, and a mood of the person on the display screen.

10. The step of detecting the first convergence of the eyes of the first user includes:

2. The method of gaze alignment of claim 1, further comprising detecting the first convergence of the eyes of the first user using at least one of an inertial measurement unit (IMU), a depth sensor, a light detection and ranging (LiDar) sensor, a near infrared (NIR) sensor, and a spatial audio detector.

11. Determining a location of an object of interest of a first user; receiving a first one or more images of the first user captured by a camera at a camera position different from the position of the object of interest; Detecting a first convergence of an eye of the first user; Calculating the first convergence disparity for viewing the object of interest; performing gaze correction of the first one or more images based on the inconsistency of the first convergence to view the object of interest.

23. An electronic system comprising: a processing circuit configured to:

12. The processing circuitry includes: receiving a second one or more images of a second user engaging in immersive telepresence with the first user; deriving an interpupillary position of the second user from the second one or more images; displaying the second image or images with the interpupillary position set to a screen plane of a display screen of the first user; determining the interpupillary position in the screen plane of the display screen as the position of the object of interest of the first user; 12. The electronic system of claim 11, configured to:

13. The processing circuitry includes: determining a center point of the display screen of the first user as the location of the object of interest of the first user; 12. The electronic system of claim 11, configured to:

14. The processing circuitry includes: receiving a second one or more images of a second user engaging in immersive telepresence with the first user; displaying the second image or images on a display screen of the first user; determining a location on the display screen for displaying the second user's eye as the location of the object of interest of the first user; 12. The electronic system of claim 11, configured to:

15. The processing circuitry includes: Sense the first convergence based on an eye tracking sensor separate from the camera; or Detecting the first congestion based on an image analysis of the first one or more images of the first user.

12. The electronic system of claim 11, configured to:

16. The processing circuitry includes: Calculating the first convergence of the eyes of the first user based on a head position of the first user.

12. The electronic system of claim 11, configured to:

17. The processing circuitry includes: determining a point of gaze according to the first convergence of the eyes of the first user and a head position of the first user; Calculating the mismatch of the point of gaze relative to the location of the object of interest of the first user.

12. The electronic system of claim 11, configured to:

18. The processing circuitry includes: modifying a head position of the first user in the first one or more images; or modifying a body position of the first user in the first one or more images; or correcting a head pose of the first user in the first one or more images; or Modifying a body pose of the first user in the first one or more images; or Modifying the head rotation angle of the first user in the first one or more images; or Modifying a body rotation angle of the first user in the first one or more images.

12. The electronic system of claim 11, configured to:

19. The processing circuitry includes: determining the location of the object of interest according to at least one of a facial expression of a person on a display screen, a visual emotion analysis of the person on the display screen, and a mood of the person on the display screen; 12. The electronic system of claim 11, configured to:

20. The processing circuitry includes: Detecting the first convergence of the eyes of the first user using at least one of an inertial measurement unit (IMU), a depth sensor, a light detection and ranging (LiDar) sensor, a near infrared (NIR) sensor, and a spatial audio detector.

12. The electronic system of claim 11, configured to:

Images