Three-dimensional video communication method having highly-sensitive motion parallax
By acquiring and predicting eye pose information in 3D video communication, generating and transmitting images from future perspectives, the problem of motion parallax lag is solved, achieving a highly sensitive 3D video communication experience suitable for cloud computing environments.
Patent Information
- Application Number
- PCT/CN2024/136678
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-04-29
- Filing Date
- 2024-12-04
- Publication Date
- 2026-01-02
AI Technical Summary
In existing 3D video communication, the latency in data acquisition, data transmission, 3D reconstruction, and rendering calculations causes motion parallax lag, and existing methods such as increasing sensor equipment and bandwidth have not completely solved this problem.
By collecting eye pose information on the local user side, using eye tracking and prediction modules, combined with AI estimation or geometric algorithms to obtain eye pose, and using time series prediction algorithms or state estimation algorithms to predict future eye movements, the system generates and transmits images from the future perspective, compensating for latency.
It achieves highly sensitive motion parallax 3D video communication, enabling users to more accurately perceive the movement and spatial position of remote participants, providing an immersive experience, reducing latency, and making it suitable for cloud computing scenarios.
Smart Images
Figure CN2024136678_02012026_PF_FP_ABST
Abstract
Description
Method for three-dimensional video communication with high sensitivity motion parallax
[0001] Related applications
[0002] The present application claims priority to the Chinese patent application No. 2024105260456, filed on April 29, 2024, entitled "Method for three-dimensional video communication with high sensitivity motion parallax", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD
[0003] The present application belongs to the field of computer, and particularly relates to a method for three-dimensional video communication with high sensitivity motion parallax. BACKGROUND
[0004] With the continuous development of network, AI (Artificial Intelligence) and cloud computing, three-dimensional video communication has become the mainstream way of future remote communication. Compared with traditional two-dimensional, three-dimensional video communication captures and transmits three-dimensional images and environmental information of remote participants, enabling local users to more accurately perceive the motion, depth and spatial position of the other party. It not only makes up for the obstacles caused by geographical distance, but also provides a more immersive experience, promoting real-time cooperation and communication. In order to achieve the 3D (Three-Dimensional) effect of watching, binocular disparity, motion parallax and depth parallax need to be considered. Among them, motion parallax is the parallax effect based on the motion of the participant, and by capturing the eye position of the local user, the 3D display device displays the picture of the remote participant at this viewing angle. However, due to the time delay of data acquisition, data transmission, three-dimensional reconstruction and rendering, the eye movement of the local user will cause the picture switching at this viewing angle to have a time delay, that is, the motion parallax has a lag phenomenon. In view of this problem, the common method in the industry is to increase sensor devices and increase bandwidth to reduce the time delay of the picture, but these methods cannot completely solve the lag problem of motion parallax. There is a lack of a method for three-dimensional video communication with high sensitivity motion parallax in the prior art. SUMMARY
[0005] The present application provides a method for three-dimensional video communication with high sensitivity motion parallax, comprising:
[0006] Step 1, at the local user side, the picture captured by the acquisition device is used for eye tracking, and the pose information of the eye of the local user in space is obtained through the eye tracking module;
[0007] Step 2, the picture captured by the camera at the remote participant side is used for three-dimensional reconstruction and rendering;
[0008] Step 3, the spatial pose information of the local user's eyes and the picture data of the remote participant are transmitted to the cloud host, and the three-dimensional reconstruction rendering module generates a new perspective image according to the eye pose and picture information;
[0009] Step 4, the new perspective image is transmitted to the local user for completion.
[0010] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the application, step 1 includes: using an eye tracking module to obtain the pose information of the local user's eyes in space by AI estimation or geometric algorithm.
[0011] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the application, the eye tracking module uses AI estimation or geometric algorithm to obtain the pose information of the local user's eyes in space, including:
[0012] Step 1.1, initialize the positions of the two cameras, calculate the relative position relationship between the two cameras, use camera calibration algorithm to obtain camera internal and external parameters according to the camera shooting picture, or combine depth sensor to solve in three-dimensional space or use artificial intelligence algorithm to estimate the relative position;
[0013] Step 1.2, face recognition and face key point detection are performed on the pictures shot by the two cameras respectively, the corresponding feature points are extracted, and the pixel coordinates of each feature point in the picture and the corresponding relationship of the feature points in the two cameras are obtained;
[0014] Step 1.3, according to the relative relationship between the two cameras and the feature points in the picture, determine a unique spatial triangle, and calculate the position coordinates of the eyes in three-dimensional space by geometric solving method;
[0015] Step 1.4, combine the face features and head pose to calculate the line of sight direction of the two eyes, and obtain the pose of the eyes in three-dimensional space.
[0016] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the application, in step 1, an eye pose prediction module is added to compensate for the time delay of collection, transmission and calculation by predicting the eye trajectory at future time and the eye pose at future time, and solve the hysteresis phenomenon of motion parallax.
[0017] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the application, the eye pose prediction module is added, including:
[0018] The local user side camera collects pictures, and the eye tracking module obtains the pose information of the local user's eyes in space;
[0019] An eye pose prediction module is added to estimate the trajectory and position of eye movement by time series prediction algorithm or state estimation algorithm, and the predicted future time stamp and corresponding pose information are sent to the reconstruction and rendering module.
[0020] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, the data transmission protocol of eye movement tracking and prediction needs to be customized, including time stamp and pose information, and the format is as follows: {(ts_curr, pos_curr), (ts_pred1, pos_pred1), (ts_pred2, pos_pred2),…}
[0021] Wherein, ts_curr: current time stamp; pos_curr: current eye pose; ts_pred: future time stamp, wherein the number indicates multiple predictions; pos_pred: predicted eye pose, the number indicates multiple predictions.
[0022] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, the reconstruction and rendering module will receive multiple predicted eye poses, generate multiple new perspective pictures, and transmit the pictures back to the local user. The transmission protocol needs to be customized, including time stamp, pose information, and frame image information, and the format is as follows: {(ts_curr, pos_curr, frame_curr), (ts_pred1, pos_pred1, frame_pred1), (ts_pred2, pos_pred2, frame_pred2),…}
[0023] Wherein, frame represents the picture frame of the new perspective generated at different time according to different eye poses.
[0024] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, in step 1, according to the requirements of the eye movement tracking algorithm, the acquisition device uses a camera, and the sampling camera uses a single or multiple cameras.
[0025] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, in step 2, according to the requirements of the reconstruction algorithm, the acquisition device uses a camera, and the sampling camera uses a single or multiple cameras.
[0026] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, in step 2, the sampling camera uses an RGB-D camera.
[0027] As a further scheme of the high-sensitivity motion parallax three-dimensional video communication method of the present application, in step 1, the sampling camera uses an RGB camera. BRIEF DESCRIPTION OF DRAWINGS
[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description only represent some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the disclosed drawings.
[0029] FIG. 1 is a schematic diagram of a process that a local user needs to go through to see the 3D effect of a remote participant in three-dimensional video communication;
[0030] FIG. 2 is a schematic diagram of an eye pose prediction module according to some embodiments of the present application. DETAILED DESCRIPTION
[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.
[0032] The technical solutions of the present application will be further described in detail below with reference to the drawings:
[0033] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application. The present application will be described in detail below according to the drawings and embodiments, and the purposes and effects of the present application will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.
[0034] The main purpose of the present application is to track the eye pose by computer vision algorithm, predict the motion trajectory and future pose of the eye in space, transmit the predicted eye pose information to the three-dimensional reconstruction and rendering module, calculate the picture image under the future view angle, achieve the effect of delay compensation in collection, transmission and calculation, and realize high-sensitivity motion parallax three-dimensional video communication.
[0035] With the continuous development of network, AI, and cloud computing, three-dimensional video communication has become the mainstream way of future remote communication. Compared with traditional two-dimensional, three-dimensional video communication captures and transmits three-dimensional images and environmental information of remote participants, enabling local users to more accurately perceive the motion, depth, and spatial position of the other party. It not only makes up for the obstacles caused by geographical distance, but also provides a more immersive experience, promoting real-time cooperation and communication. In order to achieve the 3D effect of viewing, binocular disparity, motion parallax, and depth parallax need to be considered. Motion parallax is the parallax effect based on the motion of the participant, and by capturing the eye position of the local user, the 3D display device displays the image of the remote participant at that viewing angle. However, due to the time delay of data acquisition, data transmission, three-dimensional reconstruction, and rendering calculations, the eye movement of the local user will cause a time delay in the switching of the image at that viewing angle, that is, the motion parallax has a lag phenomenon.
[0036] In three-dimensional video communication, the local user needs to see the 3D effect of the remote participant, which needs to go through the following steps, as shown in FIG. 1. First, the camera at the local user side captures the image for eye tracking. According to the eye tracking algorithm requirements, the acquisition camera can be single or multiple, and the eye tracking module obtains the pose information (pose includes position and direction) of the local user's eye in space through AI estimation or geometric algorithm; the scheme mainly tracks the eye pose, uses eye pose prediction to compensate for the time delay in the new view generation process, the steps of the eye tracking module here are just an example of a feasible means to achieve this function, other methods to achieve the function of this module can be used; the scheme uses eye pose prediction to compensate for the time delay in the new view generation process, and the lag phenomenon of motion parallax is more obvious due to the increase in transmission time delay of cloud service, but the scheme can also have optimization effect in the end-to-end mode, here only the cloud service case is used to illustrate.
[0037] At the same time, the camera at the remote participant side captures the image for three-dimensional reconstruction and rendering. According to the reconstruction algorithm requirements, the acquisition camera can be single or multiple, or even an RGB-D (Red Green Blue-Depth) camera. The acquisition device includes but is not limited to a camera, and any sensor that can obtain relevant information can be used. It is not limited to an RGB-D camera, an RGB (Red Green Blue) camera can also be used according to the algorithm requirements of three-dimensional reconstruction or new view generation, or any sensor that can obtain the required data for the algorithm can be used.
[0038] Then, due to the limited computing power at the end side, cloud service can be used to fully utilize the computing power. The spatial pose information of the local user's eye and the image data of the remote participant need to be transmitted to the cloud host, and the three-dimensional reconstruction and rendering module generates a new view image based on the above-mentioned eye pose and image information. Finally, the new view image is transmitted to the local user for completion.
[0039] Wherein the eye movement tracking module can use the following scheme to initialize the dual-camera position, calculate the relative position relationship between the dual cameras, use the camera calibration algorithm to obtain the camera internal and external parameters according to the camera shooting picture, or solve in three-dimensional space combined with the depth sensor or estimate the relative position using an artificial intelligence algorithm. The pictures taken by the dual cameras are subjected to face recognition and face key point detection respectively, the corresponding facial feature points are extracted, and the pixel coordinates of each feature point in the picture and the corresponding relationship of the feature points in the two cameras are obtained. According to the relative relationship between the dual cameras and the feature points in the picture, a unique spatial triangle is determined, and the position coordinates of the eye in the three-dimensional space are calculated using a geometric solving method. The line-of-sight direction of the eyes is calculated in combination with the facial features and the head pose, and the pose of the eyes in the three-dimensional space is obtained.
[0040] However, the inevitable time delay in the above scheme process will affect the real-time motion parallax effect in three-dimensional video communication. When the local user moves at t1, the camera of the local user captures the movement of the eye of the local user, and after the time delay of data acquisition, data transmission, reconstruction and rendering calculation and picture return, the picture is seen at t2, that is, the picture of the current view can be seen after the eye movement (t2-t1) time. This is the reason for the lag phenomenon of motion parallax.
[0041] In order to solve the sensitivity problem of motion parallax in three-dimensional video communication, an eye pose prediction module is added to compensate for the time delay of acquisition, transmission and calculation by predicting the future eye trajectory and the eye pose at a future time, thereby fundamentally solving the lag phenomenon of motion parallax. The specific steps of a three-dimensional video communication method with high sensitivity motion parallax according to an embodiment of the application are shown in FIG. 2.
[0042] First, as in the previous steps, the camera of the local user side acquires the picture, and the eye movement tracking module obtains the pose information of the eye of the local user in space. Then, an eye pose prediction module is added to estimate the trajectory and position of the eye movement through a time series prediction algorithm or a state estimation algorithm, and to send the predicted future time stamp and the corresponding pose information to the reconstruction and rendering module. In order to solve the accuracy problem of the prediction algorithm, multiple future eye poses can be predicted. The data transmission protocol of eye movement tracking and prediction needs to be customized here, mainly including time stamp and pose information, and the format is as follows: {(ts_curr, pos_curr), (ts_pred1, pos_pred1), (ts_pred2, pos_pred2),…}
[0043] Wherein, ts_curr: current time stamp; pos_curr: current eye pose; ts_pred: future time stamp (numbered to predict multiple); pos_pred: predicted eye pose (numbered to predict multiple).
[0044] Thus, the reconstruction and rendering module receives multiple predicted eye poses, generates multiple new perspective pictures, and transmits the pictures back to the local user. The transmission protocol needs to be customized and mainly includes a timestamp, a pose information, and a frame image information, and has the following format: {(ts_curr, pos_curr, frame_curr), (ts_pred1, pos_pred1, frame_pred1), (ts_pred2, pos_pred2, frame_pred2), …}
[0045] Among them, frame represents the picture frame of the new perspective generated according to different eye poses at different times. These data are transmitted back to the local user, and at this time, the time has reached t2, and the picture with the closest eye pose to the predicted eye pose at t2 is selected from the above set for display.
[0046] The following describes the whole process of high-sensitivity motion parallax achieved by the scheme by taking the prediction of a pose at a time as an example:
[0047] The local user is in a motion state, and the time at t1 starts timing. The time at which the camera captures a picture is recorded as ts_curr, and the eye pose corresponding to the picture is calculated by an eye tracking algorithm and recorded as pos_curr. The eye pose at a future ts_pred is predicted by an eye position prediction module as pos_pred. The transmission timestamp and eye pose data are transmitted, the reconstruction and rendering module generates a new perspective picture corresponding to pos_pred, and the predicted picture is transmitted to the local user for viewing. At this time, it has reached t2, and since the result pos_pred of the eye position prediction module is the picture of the corresponding perspective at t2, (t2-ts_pred) is approximately 0. Therefore, the motion parallax lag problem is solved by predicting to compensate for the acquisition, transmission, and calculation time delay.
[0048] In summary, the present application aims to achieve high-sensitivity motion parallax three-dimensional video communication. By tracking the eye pose and predicting the motion trajectory and future pose, the picture of the future perspective is reconstructed and rendered, so as to compensate for the acquisition, transmission, and calculation time delay. The user can more accurately perceive the stereoscopic effect in three-dimensional communication and provide an immersive viewing experience.
[0049] 1. The method tracks and predicts the eye pose, transmits information to the three-dimensional reconstruction and rendering module, generates a picture at a future perspective, achieves high-quality three-dimensional video communication, and provides a more immersive and realistic experience.
[0050] 2. The method considers the acquisition, transmission, and calculation time delay in the processing process, achieves the effect of time delay compensation by predicting the eye movement, fundamentally solves the motion parallax lag problem, and improves the user experience.
[0051] 3. The method does not require adding sensor devices, increasing bandwidth, is highly portable, is suitable for cloud computing scenarios, fully utilizes cloud computing power, and is a general method for reducing latency.
[0052] Those skilled in the art can understand that the above description is only examples of the application and is not intended to limit the application, although the application is described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions recorded in the foregoing examples or make equivalent replacements for part of the technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.
[0053] Finally, it should be noted that: the above description is only an embodiment of the present application and is not intended to limit the present application, although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or make equivalent replacements for part of the technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.
[0054] The technical features of the above-described embodiments can be combined arbitrarily, and in order to make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope recorded in the present description.
[0055] The above-described embodiments only express several implementation manners of the present application, the description is more specific and detailed, but it should not be understood as a limitation on the patent application scope. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application patent should be subject to the appended claims.
Claims
1. A highly sensitive three-dimensional video communication method with high motion parallax, comprising: Step 1: On the local user side, the images captured by the acquisition device are used for eye tracking, and the pose information of the local user's eyes in space is obtained through the eye tracking module. Step 2: Capture images using cameras on the remote participant's side for 3D reconstruction and rendering; Step 3: Transfer the spatial pose information of the local user's eyes and the image data of the remote participant to the cloud host. The 3D reconstruction and rendering module generates a new perspective image based on the eye pose and image information. Step 4: Transmit the new perspective image to the local user to complete the process.
2. The method according to claim 1, wherein, Step 1 includes: using an eye-tracking module to obtain the pose information of the local user's eyes in space through AI estimation or geometric algorithms.
3. The method according to claim 2, wherein an eye-tracking module is used to obtain the pose information of the local user's eyes in space through AI estimation or geometric algorithms, comprising: Step 1.1: Initialize the dual-camera positions, calculate the relative positional relationship between the two cameras, and use the camera calibration algorithm to obtain the camera intrinsic and extrinsic parameters based on the images captured by the cameras, or combine the depth sensor to solve in three-dimensional space or use artificial intelligence algorithms to estimate the relative positions. Step 1.2: Perform face recognition and face key point detection on the images captured by the dual cameras respectively, extract the corresponding facial feature points, and obtain the pixel coordinates of each feature point in the image and the correspondence between the feature points in the two cameras. Step 1.3: Based on the relative relationship between the two cameras and the feature points in the image, determine a unique spatial triangle and use a geometric solution method to calculate the position coordinates of the eye in three-dimensional space. Step 1.4: Combine facial features and head pose to calculate the gaze direction of the eyes and obtain the pose of the eyes in three-dimensional space.
4. The method according to claim 1, wherein, In step 1, an eye pose prediction module is added. By predicting the eye trajectory at future times and the eye pose at a certain future moment, the acquisition, transmission, and calculation delays are compensated to solve the hysteresis phenomenon of motion parallax.
5. The method according to claim 4, wherein, Added an eye pose prediction module, including: The local user-side camera captures images, and the eye-tracking module obtains the pose information of the local user's eyes in space. An eye pose prediction module has been added. This module estimates the trajectory and position of eye movements using a time-series prediction algorithm or a state estimation algorithm, and sends the predicted future timestamp and corresponding pose information to the reconstruction and rendering module.
6. The method according to claim 5, wherein, The data transmission protocol for eye tracking and prediction needs to be customized, including timestamps and pose information, in the following format: {(ts_curr,pos_curr), (ts_pred1,pos_pred1), (ts_pred2,pos_pred2),…} Where ts_curr: the current timestamp; pos_curr: the current eye pose; ts_pred: the future timestamp, where the number indicates multiple predictions; pos_pred: the predicted eye pose, where the number indicates multiple predictions.
7. The method according to claim 6, wherein, The reconstruction and rendering modules receive multiple predicted eye poses, generate multiple new viewpoint images, and transmit these images back to the local user. The transmission protocol needs to be customized and includes timestamps, pose information, and frame image information, in the following format: {(ts_curr,pos_curr,frame_curr), (ts_pred1,pos_pred1,frame_pred1), (ts_pred2,pos_pred2,frame_pred2),…} Here, frame represents a frame of the image with a new perspective generated at different times based on different eye poses.
8. The method according to claim 1, wherein, In step 1, according to the requirements of the eye-tracking algorithm, the acquisition device is a camera, and the sampling camera can be one or more.
9. The method according to claim 1, wherein, In step 2, according to the reconstruction algorithm requirements, the acquisition device is a camera, and the acquisition camera can be a single camera or multiple cameras.
10. The method according to claim 8, wherein, In step 2, the acquisition camera is an RGB-D camera.
11. The method according to claim 9, wherein, In step 1, the acquisition camera is an RGB camera.