Holographic communication device, system, method and storage medium

Abstract

The present disclosure provides a holographic communication device, system, method, and storage medium, belonging to the field of computer vision technology. The holographic communication device of the present disclosure includes a processor configured to determine the three-dimensional coordinates of a user's human eyes in real time based on multiple received first images; decode the received video stream into multiple second images; the multiple second images are images of a target object captured by different second cameras from different perspectives; based on the sequence information of each second image, calculate the disparity between the perspectives of two adjacent second images to obtain multiple disparity maps, and convert each disparity map into point cloud data; based on each point cloud data, convert the image features of the second image into three-dimensional coordinates of the human eyes to obtain a viewpoint feature map; use a pre-trained rendering neural network model to render the viewpoint feature map to obtain new viewpoint maps corresponding to the user's two eyes; and the holographic display is configured to perform holographic display based on each new viewpoint map.

Description

Technical Field

[0001] The present disclosure belongs to the field of computer vision technology, and in particular relates to a holographic communication device, system, method and storage medium. Background Art

[0002] Holographic communication utilizes optical and computer image processing techniques to remotely capture images of a target object and its surroundings, transmitting them over a network to a terminal for real-time, three-dimensional display. Compared to traditional two-dimensional image communication, holographic communication offers a more authentic and realistic communication experience, and has broad application prospects in remote video, audio and video entertainment, online education, healthcare, and other fields. Summary of the Invention

[0003] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provide a holographic communication device, system, method and storage medium.

[0004] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a holographic communication device, which includes a holographic display, a processor, and a memory; the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the processor performs operations, including:

[0005] Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images, wherein the multiple first images are images of the user captured by multiple first cameras from different perspectives;

[0006] receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device;

[0007] Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data;

[0008] For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image into the three-dimensional coordinates of the human eye to obtain a viewpoint feature map;

[0009] Rendering the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's binocular eyes;

[0010] The holographic display is configured to perform holographic display based on each of the new viewpoint images.

[0011] In some embodiments, the operation of calculating the inter-view disparity of two adjacent second images based on the sequence information of each second image to obtain multiple disparity maps includes:

[0012] Based on the sequence information of each second image, a pre-trained disparity estimation model is used to calculate the disparity between the perspectives of two adjacent second images to obtain multiple disparity maps.

[0013] In some embodiments, the operation of converting each of the disparity maps into point cloud data includes:

[0014] For any of the disparity maps, convert the disparity map into a depth map based on parameter information of the second camera corresponding to the disparity map;

[0015] Based on the parameter information of the second camera and the position information of the foreground point in the second image corresponding to the depth map, the depth map is converted from the image two-dimensional coordinate system to the human eye three-dimensional coordinate system to obtain a point cloud set; the point cloud set includes point cloud data converted based on one of the depth maps.

[0016] In some embodiments, the depth map is determined using the following formula 1:

[0017] Formula 1: Among them, f x represents the focal length of the first camera in the X-axis direction of the second camera used to capture the selected second image in the two-dimensional coordinate system of the image; B represents the baseline of the second camera used to capture the selected second image; d represents the grayscale value of a pixel point in the disparity map; and D represents the depth value of the corresponding pixel point in the disparity map.

[0018] In some embodiments, the following expression 2 is used to determine a point cloud data in the point cloud set;

[0019] Formula 2: Where p represents a point cloud data; x, y, z represent the coordinates of the foreground point in the three-dimensional coordinate system of the human eye; x′, y′ represent the coordinates of the foreground point in the two-dimensional coordinate system of the image; f x ,f y They respectively represent the second camera used to capture the selected second image, the first camera focal length in the X-axis direction and the second camera focal length in the Y-axis direction in the two-dimensional coordinate system of the image.

[0020] In some embodiments, the operation of converting the image features of the second image into the three-dimensional coordinates of the human eye based on each of the point cloud data to obtain a viewpoint feature map includes:

[0021] For each of the point cloud data, regressing a tensor corresponding to a Gaussian distribution of the point cloud data using a convolutional neural network based on image features of the second image;

[0022] Obtaining Gaussian point cloud data corresponding to the point cloud data based on a tensor corresponding to the Gaussian distribution of the point cloud data;

[0023] The Gaussian point cloud data are rasterized to the three-dimensional coordinates of the human eye using a Gaussian splattering algorithm to obtain a viewpoint feature map.

[0024] In some embodiments, the following formula 3 is used to determine the Gaussian point cloud data corresponding to the point cloud data:

[0025] Formula 3: Where Σ=RSS T R T Represents the intermediate result of the matrix operation, R represents the quaternion of the rotation parameter in the tensor, S represents the scale in the tensor, () T Represents the matrix transpose operation; G(p) represents the Gaussian point cloud data corresponding to the point cloud data; p represents the point cloud data.

[0026] In some embodiments, the rendering neural network model includes a UNet convolutional neural network with a residual network structure.

[0027] In some embodiments, the operation further includes training the disparity estimation model, specifically including:

[0028] Based on sequence information of each sample image in a training data set and the virtual object to which each sample image belongs, selecting a target sample image of the virtual object at a set of adjacent viewing angles from the training data set; the training data set includes sample images of different virtual objects captured by a virtual camera at different viewing angles;

[0029] Inputting a set of the target sample images into the disparity estimation model to calculate the inter-view disparity and output a theoretical disparity map;

[0030] A loss function is constructed based on the theoretical disparity map and a preset real disparity map, and the loss function is used for backpropagation to continuously train the disparity estimation model; target sample images of the virtual object under another set of adjacent perspectives are continuously selected from the training data set, or target sample images of other virtual objects under a set of adjacent perspectives are selected from the training data set, and the process of inputting a set of the target sample images into the disparity estimation model to calculate the disparity between perspectives and outputting the theoretical disparity map is returned to execute until the loss function converges to obtain a trained disparity estimation model.

[0031] In some embodiments, the operation further includes training the rendering neural network model, specifically including:

[0032] Get the pre-rendered theoretical depth map;

[0033] Based on the parameter information of the virtual camera, the theoretical depth map is converted from the sample image two-dimensional coordinate system to a virtual three-dimensional coordinate system to obtain a training point cloud set; the training point cloud set includes theoretical point cloud data obtained based on the conversion of the theoretical depth map;

[0034] For each of the theoretical point cloud data, based on the image features of the target sample image, using a convolutional neural network, regressing the tensor corresponding to the Gaussian distribution of the theoretical point cloud data;

[0035] Obtain theoretical Gaussian point cloud data corresponding to the theoretical point cloud data based on a tensor corresponding to the Gaussian distribution of the theoretical point cloud data;

[0036] Using a Gaussian splattering algorithm, rasterizing each of the theoretical Gaussian point cloud data to a target viewing angle of the virtual object to obtain a theoretical viewpoint feature map;

[0037] Rendering the theoretical viewpoint feature map using the rendering neural network model to be trained to obtain theoretical viewpoint maps corresponding to the binocular eyes of the virtual object;

[0038] A loss function is constructed based on the theoretical viewpoint map and a preset real viewpoint map, and the loss function is used for backpropagation to continuously train the rendering neural network model; and the step of obtaining the pre-rendered theoretical depth map is returned and executed sequentially until the loss function converges to obtain a trained rendering neural network model.

[0039] In a second aspect, an embodiment of the present disclosure further provides a holographic communication system, comprising a plurality of first cameras and a holographic communication device; the holographic communication device comprises a holographic display, a processor, and a memory;

[0040] a plurality of the first cameras, configured to capture first images of the user from different perspectives respectively, and send the first images to the processor;

[0041] The memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the processor performs operations, the operations including:

[0042] Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images, wherein the multiple first images are images of the user captured by multiple first cameras from different perspectives;

[0043] receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device;

[0044] Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data;

[0045] For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image to the user's perspective corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map;

[0046] Rendering the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's binocular eyes;

[0047] The holographic display is configured to perform holographic display based on each of the new viewpoint images.

[0048] In some embodiments, the first camera is a color camera.

[0049] In some embodiments, the plurality of said first cameras includes four said first cameras.

[0050] In a third aspect, the embodiments of the present disclosure further provide a holographic communication method, which is applied to the holographic communication device as described in the first aspect; wherein the holographic communication method includes:

[0051] Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images, wherein the multiple first images are images of the user captured by multiple first cameras from different perspectives;

[0052] receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device;

[0053] Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data;

[0054] For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image to the user's perspective corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map;

[0055] The viewpoint feature map is rendered using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's two eyes, and then holographically displayed.

[0056] In a fourth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the holographic communication method described in the third aspect are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a holographic communication device provided in an embodiment of the present disclosure;

[0058] Figure 2 A flowchart of a processor provided for an embodiment of the present disclosure configured to perform operations related to holographic communication;

[0059] Figure 3 A schematic diagram of a holographic communication system provided by an embodiment of the present disclosure;

[0060] Figure 4 A flow chart of a holographic communication method provided in an embodiment of the present disclosure;

[0061] Figure 5 A schematic diagram of an interaction method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0063] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "an" or "the" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0064] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0065] In the related art, traditional holographic communication systems generally require the use of relatively complex image acquisition equipment, such as dense camera arrays (for example, more than 100 SLR cameras synchronously set up), or high-precision depth cameras, etc. Such image acquisition equipment is expensive, and there is no shortage of customized hardware, which is difficult to popularize. However, using sparse color cameras (also known as RGB cameras) as image acquisition devices for holographic communication systems makes it difficult to obtain sufficient three-dimensional information from users, resulting in poor holographic display effects. Therefore, holographic communication using sparse RGB cameras as image acquisition devices is currently more difficult. In addition, due to the high real-time requirements of holographic communication systems, a large number of algorithms that can only perform offline reconstruction and rendering are no longer applicable, further increasing the difficulty of implementing holographic communication.

[0066] In view of this, embodiments of the present disclosure provide a holographic communication device, system, method, and storage medium that substantially eliminate one or more of the problems caused by limitations and defects of the related art.

[0067] The following first describes the holographic communication device. The holographic communication device provided in the embodiments of the present disclosure is configured to provide a holographic stereoscopic display of a target object provided by another device with which the holographic communication device is communicating. The other device may be a device that remotely communicates with the holographic communication device, specifically a holographic communication device identical to the holographic communication device provided in the embodiments of the present disclosure, or a communication device that does not have a holographic display function.

[0068] Figure 1 A schematic diagram of a holographic communication device provided in an embodiment of the present disclosure, such as Figure 1 As shown, it includes a holographic display 101, a processor 102 and a memory 103; the memory 103 stores machine-readable instructions executable by the processor 102, and when the machine-readable instructions are executed by the processor 102, the processor 102 performs operations. Figure 2 The processor 102 provided in the embodiment of the present disclosure is configured to execute a flowchart of operations related to holographic communication, such as Figure 2 As shown, the operations performed by the processor 102 include the following steps S11 to S16.

[0069] S11 . Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images.

[0070] In this step, the multiple first images are images of the user captured by multiple first cameras from different perspectives, specifically images of the upper body or face of the user. The user refers to a user using the holographic communication device 100. In the actual application stage, the user is located in front of the holographic display 101 of the holographic communication device 100. The multiple first cameras can be color cameras (i.e., RGB cameras) arranged on the holographic display 101. The specific number of settings is between 2 and 4. This disclosure takes 4 RGB cameras as an example for explanation. Different first cameras have different positions and different shooting angles relative to the user, so they can capture the first images of the user at different perspectives. The first camera is configured to send the captured first image to the processor 102. Each first image contains a picture of the same user.

[0071] The processor 102 is configured to combine the image features of the user in each first image to calculate the three-dimensional coordinates of the human eye of the same user. Here, the three-dimensional coordinates of the human eye include the left eye coordinates and the right eye coordinates in the three-dimensional coordinate system of the human eye.

[0072] S12: Receive a video stream, and decode the video stream into multiple second images.

[0073] Among them, the multiple second images are images of the target object taken by different second cameras from different perspectives; wherein the target object is a physical object in the operating environment of other devices, for example, it can be a person or an object; the other device is a communication device that communicates with the holographic communication device 100; the second camera communicates with other devices. The multiple second cameras can be color cameras (i.e., RGB cameras) set on other devices, and the specific number of settings is between 2 and 4. The present disclosure takes 4 RGB cameras as an example for explanation. The second camera is configured to send the captured second image to other devices, and the other devices encode the multiple second images into a video stream and send it to the holographic communication device 100 side. Among them, the encoding method can be H.265 encoding, which is a high-efficiency video coding (HEVC) method. The video stream not only contains multiple encoded second images, but also contains sequence information corresponding to each second image, such as the perspective of the corresponding second camera.

[0074] The processor 102 is configured to receive a video stream and decode the video stream into a second image of a target object under multiple perspectives.

[0075] S13 . Based on the sequence information of each second image, respectively calculate the inter-view disparity of two adjacent second images to obtain a plurality of disparity maps.

[0076] Taking the four second images captured by the four second cameras as an example, the sequence numbers indicated by their respective sequence information are 1, 2, 3, and 4, respectively; wherein, sequence numbers 1 and 2 are a group of adjacent second images, sequence numbers 2 and 3 are a group of adjacent second images, and sequence numbers 3 and 4 are a group of adjacent second images.

[0077] Specifically, based on the sequence information of each second image, a pre-trained disparity estimation model is used to calculate the inter-view disparity of two adjacent second images, thereby obtaining multiple disparity maps. The disparity estimation model integrates a disparity estimation algorithm. Exemplarily, the disparity estimation algorithm is used to calculate the inter-view disparity of two adjacent second images, thereby obtaining a disparity map. For second images with sequence numbers 1, 2, 3, and 4, three disparity maps are ultimately obtained.

[0078] S14. Convert each disparity map into point cloud data.

[0079] In this step, a disparity map is converted into a point cloud set; the point cloud set includes multiple point cloud data; wherein the point cloud data is the point cloud of the second image corresponding to the disparity map at the foreground point position. For a second image, an image segmentation algorithm is used to obtain the position information of the foreground point and the background point corresponding to the second image. The position information of the foreground point is the coordinate of the foreground point in the image's two-dimensional coordinate system; the position information of the background point is the coordinate of the background point in the image's two-dimensional coordinate system. Once the position information of the foreground point is known, the pixel point at the foreground point position in the disparity map can be determined, and then the point cloud data corresponding to the pixel point at the foreground point position can be determined to form a point cloud set.

[0080] Here, the foreground points may be understood as points belonging to the 3D target object in the second image, and the background points may be understood as points not belonging to the target object, such as background pixels or environment pixels.

[0081] S15. For any disparity map, based on each point cloud data, convert the image features of the second image into three-dimensional coordinates of the human eye to obtain a viewpoint feature map.

[0082] Specifically, for each point cloud data point, based on the image features of the second image, a tensor corresponding to the Gaussian distribution of the point cloud data is predicted. This tensor includes, but is not limited to, a quaternion representing the rotation parameters of the point cloud data, the scale of the point cloud data, and the transparency of the point cloud data. Next, a Gaussian distribution of the point cloud data point is fitted based on this tensor to obtain the Gaussian point cloud data corresponding to the point cloud data point. The Gaussian point cloud data point is then converted to three-dimensional coordinates for the human eye, meaning the image features of the second image are converted to the user's perspective to obtain a viewpoint feature map.

[0083] Since the three-dimensional coordinates of the human eye include left eye coordinates and right eye coordinates, this step is to convert and match the Gaussian point cloud data corresponding to the point cloud data to left eye coordinates and right eye coordinates respectively, and obtain the left eye viewpoint feature map matching the left eye and the right eye viewpoint feature map matching the right eye respectively, so that they correspond to the user's real perspective and prepare for three-dimensional stereoscopic viewing.

[0084] S16. Render the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's two eyes.

[0085] In this step, the rendering neural network model is a pre-trained deferred rendering model. Please refer to the following instructions for details and will not be elaborated here.

[0086] In this step, a pre-trained rendering neural network model is used to perform delayed rendering (i.e., decoding) on ​​the viewpoint feature map matching the left eye and the viewpoint feature map matching the right eye, respectively, to obtain a new left eye viewpoint map matching the user's left eye and a new right eye viewpoint map matching the right eye, both of which are related to the actual position of each eyeball in the three-dimensional coordinate system of the human eye.

[0087] The holographic display 101 is configured to perform holographic display based on each new viewpoint image. Here, the holographic display 101 projects the left-eye new viewpoint image to the user's left eye and the right-eye new viewpoint image to the user's right eye, thereby allowing the user to experience a three-dimensional stereoscopic perception without wearing any auxiliary equipment (glasses, head-mounted display, etc.), thereby achieving highly immersive holographic communication.

[0088] The holographic communication device 100 provided in the disclosed embodiment utilizes a parallax estimation algorithm to calculate three-dimensional information, and combines this algorithm with Gaussian transformation and deferred rendering neural network technology to improve rendering quality in sparse RGB camera applications, enabling users to obtain better three-dimensional visual effects without the need for peripheral auxiliary devices. Furthermore, compared to traditional light field display technology (which requires the simultaneous generation of dozens of new viewpoints), the disclosed embodiment significantly reduces the number of synthesized new viewpoints (only requiring the generation of two viewpoints for the user's eyes), allowing users to obtain a strong three-dimensional visual effect without the need for peripheral auxiliary devices, thereby enhancing the sense of presence and immersion in remote communications.

[0089] In some embodiments, step S14 includes S141 to S142.

[0090] S141 . For any disparity map, convert the disparity map into a depth map based on parameter information of a second camera corresponding to the disparity map.

[0091] Various parameter information of each second camera is known in advance, including but not limited to the camera baseline and the focal length f of the first camera in the X-axis direction in the image two-dimensional coordinate system. x , and the focal length of the second camera in the Y-axis direction f y and other parameters.

[0092] Taking three disparity maps as an example, each disparity map corresponds to a different second camera (ie, the second camera that captures the second image to determine the disparity map), and thus the parameter information corresponding to the second camera is different.

[0093] Specifically, based on the camera baseline of the second camera corresponding to the disparity map and the focal length f of the first camera x , converting the disparity map into a depth map. Taking a disparity map as an example, the following formula 1 is used to convert the disparity map into a depth map. The conversion process for other disparity maps is similar, and the repeated parts are not repeated here.

[0094] Formula 1: Among them, f x represents the focal length of the second camera used to capture the selected second image; B represents the baseline of the second camera used to capture the selected second image; d represents the grayscale value of a pixel in the disparity map; and D represents the depth value of the corresponding pixel in the disparity map. The depth values ​​of each pixel constitute a depth map.

[0095] S142 : Based on the parameter information of the second camera and the position information of the foreground point in the second image corresponding to the depth map, convert the depth map from the image two-dimensional coordinate system to the human eye three-dimensional coordinate system to obtain a point cloud set.

[0096] The image 2D coordinate system refers to the 2D coordinate system constructed based on the depth map. The human eye 3D coordinate system refers to the world coordinate system constructed based on the user's real space. A point cloud collection includes point cloud data converted from a depth map. The depth map is converted from the disparity map, meaning that the point cloud data is the point cloud at the foreground point location corresponding to the depth map. It should be noted that the foreground point location remains unchanged in the same second image.

[0097] Specifically, based on the first camera focal length f of the second camera corresponding to the depth map x and the second camera focal length f y , and the position information of the foreground point in the second image corresponding to the depth map (x ′ ,y ′ ), converting the depth map from the image's 2D coordinate system to the human eye's 3D coordinate system to obtain a point cloud set corresponding to the depth map. Taking a depth value D in a depth map as an example, the following formula 2 is used to determine a point cloud data point in the point cloud set.

[0098] Formula 2: Where p represents a point cloud data; x, y, z represent the coordinates of the foreground point in the three-dimensional coordinate system of the human eye; x′, y′ represent the coordinates of the foreground point in the two-dimensional coordinate system of the image; f x ,f y They respectively represent the second camera used to capture the selected second image, the first camera focal length in the X-axis direction and the second camera focal length in the Y-axis direction in the image two-dimensional coordinate system.

[0099] Each foreground point in the depth map (x ′ ,y ′ ) of the point cloud data p, forming the point cloud set P corresponding to the depth map = {p i |p i ∈foreground point}, which is the geometric expression of the target object.

[0100] In some embodiments, step S15 includes S151 to S153.

[0101] S151. For each point cloud data, based on the image features of the second image, use a convolutional neural network to regress the tensor corresponding to the Gaussian distribution of the point cloud data.

[0102] The second image is fed into a pre-configured convolutional neural network model, which outputs a tensor with the same spatial scale as the original second image but a different number of channels. The channels corresponding to each pixel in this tensor include the quaternion representing the rotation parameter, scale, and transparency of the point cloud data p corresponding to that pixel.

[0103] Here, taking a point cloud set as an example, the tensor output by the convolutional neural network based on the second image includes the quaternion, scale, and transparency of the rotation parameters of each point cloud data in the point cloud set.

[0104] S152. Obtain Gaussian point cloud data corresponding to the point cloud data based on the tensor of the Gaussian distribution corresponding to the point cloud data.

[0105] Based on the quaternion and scale of the rotation parameters corresponding to the point cloud data, the Gaussian point cloud data corresponding to the point cloud data is determined. Taking a point cloud data p as an example, the Gaussian point cloud data corresponding to the point cloud data is determined using the following formula 3. The calculation process for the Gaussian point cloud data corresponding to other point cloud data p is similar, and the repeated parts are not repeated here.

[0106] Formula 3: Where Σ=RSS T R T Represents the intermediate result of matrix operation, R represents the quaternion of the rotation parameter in the tensor, the specific data is a matrix, S represents the scale in the tensor, () T Represents the matrix transpose operation; G(p) represents the Gaussian point cloud data corresponding to the point cloud data p; p represents the point cloud data, and the specific data is the coordinate (x, y, z) in the three-dimensional coordinate system of the human eye, which can be seen in Formula 2.

[0107] A depth map corresponds to a point cloud set, and a point cloud set corresponds to a Gaussian point cloud data set. The Gaussian point cloud data set includes Gaussian point cloud data after the point cloud data in the point cloud set is converted.

[0108] S153. Using the Gaussian splattering algorithm, rasterize each Gaussian point cloud data to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map.

[0109] In this step, for a Gaussian power data set, the Gaussian splash algorithm is used to rasterize each Gaussian point cloud data to the three-dimensional coordinates of the human eye, specifically including the left eye coordinates and the right eye coordinates, to obtain the left eye viewpoint feature map matching the left eye coordinates and the right eye viewpoint feature map matching the right eye coordinates.

[0110] The theoretical expression of this step is: F = rasterize(G); where F represents the viewpoint feature map; G represents the Gaussian point cloud data corresponding to the entire second image; and rasterize(G) represents the rasterization algorithm.

[0111] In some embodiments, the operation of the processor 102 further includes training a disparity estimation model, specifically including S21 to S23 .

[0112] S21. Based on the sequence information of each sample image in the training data set and the virtual object to which each sample image belongs, select a target sample image of a virtual object under a set of adjacent viewing angles from the training data set.

[0113] The sequence information of the sample images can be determined based on the position information of multiple pre-arranged virtual cameras. For example, based on the position of the virtual camera, the sample images captured by it are encoded as sequence numbers 1, 2, ..., n, from left to right, where n is an integer. The training dataset includes sample images of different virtual objects captured by the virtual camera at different viewing angles. Adjacent viewing angles can be understood as viewing angles corresponding to adjacent sequence numbers. Different virtual cameras have different parameter information.

[0114] The viewpoints of the target sample images selected in this step are roughly horizontally distributed, and the parallax between viewpoints is about 10°.

[0115] During the first iteration, a group of target sample images corresponding to a virtual object is first selected; in the next iteration, a different group of target sample images corresponding to the same virtual object can be selected until the sample images of the current virtual object are traversed or the loss function converges; if the loss function does not converge, the target sample images corresponding to other virtual objects can be selected to continue the iteration, and the cycle continues until the loss function converges.

[0116] S22. Input a set of target sample images into a disparity estimation model to calculate the disparity between viewpoints, and output a theoretical disparity map.

[0117] This disparity estimation model is primarily used to perform stereo rectification on a set of target sample images, eliminating vertical disparity. Specifically, the homography required for stereo rectification is calculated based on the intrinsic and extrinsic parameters of the virtual camera corresponding to each target sample image, and bilinear interpolation is used to complete the transformation.

[0118] Optionally, the disparity estimation model includes a RAFT-Stereo algorithm, that is, a RAFT-based depth map estimation algorithm, which is mainly used for stereo matching.

[0119] S23. Construct a loss function based on the theoretical disparity map and the preset real disparity map, and use the loss function for backpropagation to continuously train the disparity estimation model.

[0120] The loss function between the theoretical disparity map and the preset real disparity map is calculated using the following formula 4:

[0121] Formula 4: Among them, d gt Represents the data corresponding to a pixel point in a preset real disparity map; d represents the data of a pixel point in a theoretical disparity map; ||1 represents a norm operation, and the norm of the difference between two points can represent the distance between the two points. Here, the operation is performed between pixels at corresponding positions. The preset real disparity map is a disparity map in a preset verification dataset. The verification dataset includes disparity maps of different virtual objects captured by a virtual camera at different perspectives. Here, the real disparity map is a disparity map captured by a virtual camera corresponding to the same perspective as the target sample image. The target sample image can be any one of a set of target sample images.

[0122] According to the above loss function The disparity estimation model is trained using the gradient descent method; thereafter, the process returns to step S21, and the target sample images of the virtual object under another set of adjacent perspectives are continuously selected from the training data set, or the target sample images of other virtual objects under a set of adjacent perspectives are selected from the training data set, and the process of S22 is returned to be executed until the loss function converges, thereby obtaining the trained disparity estimation model.

[0123] In some embodiments, the operation of the processor 102 further includes training a rendering neural network model, specifically including S31 to S37.

[0124] S31. Obtain a pre-rendered theoretical depth map.

[0125] The theoretical depth map can be converted based on the theoretical disparity map in step S22. The specific conversion process principle can be referred to step S141. For example, for each theoretical disparity map, the theoretical disparity map is converted into a theoretical depth map based on the parameter information of the virtual camera corresponding to the theoretical disparity map (such as the virtual camera baseline and focal length).

[0126] S32. Based on the parameter information of the virtual camera, each theoretical depth map is converted from the sample image two-dimensional coordinate system to the virtual three-dimensional coordinate system to obtain a training point cloud set.

[0127] The sample image 2D coordinate system refers to the 2D coordinate system constructed based on the sample image. The virtual 3D coordinate system is the world coordinate system constructed based on the virtual space where the virtual object resides. The training point cloud set includes theoretical point cloud data converted from a theoretical depth map.

[0128] Since the theoretical depth map is converted based on the theoretical disparity map, the theoretical point cloud data is the point cloud at the foreground point position corresponding to the theoretical depth map. It should be noted that the foreground point position remains unchanged under the same target sample image.

[0129] The specific principle of determining the training point cloud set in this step can be found in step S142. Finally, the point cloud data p′ of each foreground point in the theoretical depth map constitutes the point cloud set P′ corresponding to the theoretical depth map = {p ′ i |p ′ i ∈foreground point}, which is the geometric expression of the virtual object.

[0130] S33. For each theoretical point cloud data, based on the image features of the target sample image, use a convolutional neural network to regress the tensor corresponding to the Gaussian distribution of the theoretical point cloud data.

[0131] It should be noted that the technical principle of regressing the theoretical point cloud data to the tensor of Gaussian distribution in this step is the same as the technical principle of regressing the point cloud data to the tensor of Gaussian distribution in the above step S151, and the repeated parts will not be repeated.

[0132] S34. Based on the tensor of the Gaussian distribution corresponding to the theoretical point cloud data, obtain theoretical Gaussian point cloud data corresponding to the theoretical point cloud data.

[0133] It should be noted that the technical principle of determining the theoretical regression theoretical Gaussian point cloud data in this step is the same as the technical principle of determining the Gaussian point cloud data in the above step S152, and the repeated parts will not be repeated.

[0134] S35. Using the Gaussian splattering algorithm, each theoretical Gaussian point cloud data is rasterized to the target perspective of the virtual object to obtain a theoretical viewpoint feature map.

[0135] The virtual object here can be another virtual object different from the virtual object in the target sample image in S21, or it can be the same. However, when the two are the same, the target perspective in this step should be selected from different perspectives of the virtual object that is different from the corresponding perspective of the current target sample image.

[0136] In addition, the technical principle of determining the theoretical viewpoint feature map in this step is the same as the technical principle of determining the viewpoint feature map in S153, and the repeated parts are not repeated here.

[0137] S36. Render the theoretical viewpoint feature map using the rendering neural network model to be trained to obtain theoretical viewpoint maps corresponding to both eyes of the virtual object.

[0138] The rendering neural network model includes a UNet convolutional neural network with a residual network structure, which can improve the rendering accuracy by increasing the depth considerably. At the same time, the residual network can also alleviate problems such as gradient vanishing caused by increasing the depth of the rendering neural network to a certain extent.

[0139] The theoretical expression of the rendering process of this step is: in, represents the theoretical viewpoint graph; F′ represents the theoretical viewpoint feature graph; f ψ (F′) represents the rendering algorithm of the rendering neural network model.

[0140] Here, the theoretical viewpoint feature map includes a theoretical viewpoint feature map matching the left eye of the virtual object and a theoretical viewpoint feature map matching the right eye of the virtual object; the theoretical viewpoint feature map matching the left eye of the virtual object and the theoretical viewpoint feature map matching the right eye of the virtual object are respectively delayed rendered (i.e., decoded) to obtain a left eye theoretical viewpoint map matching the left eye of the virtual object and a right eye theoretical viewpoint map matching the right eye, both of which are related to the positions of each eyeball in the virtual three-dimensional coordinate system.

[0141] The model training process of S32 to S36 mentioned above is based on the same principle as the processing process of the actual application stage of the disparity estimation model, except that the input is different. One is the input sample image, which is rasterized to the target perspective of the virtual object, and the other is the input actual captured image (first image), which is rasterized to the user's current real perspective.

[0142] S37. Construct a loss function based on the theoretical viewpoint graph and the preset real viewpoint graph, and use the loss function for backpropagation to continuously train the rendering neural network model.

[0143] The loss function between the theoretical view map and the preset real view map is calculated using the following formula 5:

[0144] Formula 5: in, Represents the data corresponding to a pixel in a preset real view image; Represents the data corresponding to a pixel in the theoretical viewpoint image; Denotes the perceptual loss function. Here, operations are performed between pixels at corresponding positions. The pre-set true viewpoint image is a viewpoint image in a pre-set validation dataset. The validation dataset also includes viewpoint images of different virtual objects captured by a virtual camera at different viewing angles. Here, the true viewpoint image is a viewpoint image captured by a virtual camera at the same viewing angle as the target sample image. The target sample image can be any one of a set of target sample images.

[0145] According to the above loss function The rendering neural network model is trained using the gradient descent method; thereafter, the process returns to step S31 and loops through steps S31 to S37 until the loss function converges, thereby obtaining a trained rendering neural network model.

[0146] In some embodiments, the above-mentioned training data set is set up mainly by setting multiple virtual cameras in front of a textured high-precision human body scanning model (mainly a virtual object with a sitting posture model as the main body model). The types of virtual cameras include but are not limited to RGB cameras, depth cameras, etc. Multiple virtual cameras render RGB images and depth images (equivalent to disparity maps and viewpoint maps) of different perspectives; these RGB images serve as sample images for disparity estimation model training and sample images for rendering neural network model training; these depth maps serve as samples in the verification data set, a part of which serves as output supervision for disparity estimation model training, such as the real disparity map used for calculating the loss function of the disparity estimation model; the other part serves as output supervision for rendering neural network model training, such as the real viewpoint map used for calculating the loss function of the rendering neural network model.

[0147] In some embodiments, as Figure 1 As shown, the holographic communication device 100 further includes one or more I / O interfaces 104 ; the one or more I / O interfaces 104 are connected between the processor 102 and the memory 103 and are configured to implement information interaction between the processor 102 and the memory 103 .

[0148] In some embodiments, the processor 102 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 103 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface 104 (read-write interface) is connected between the processor 102 and the memory 103, and can realize information interaction between the processor 102 and the memory 103, including but not limited to a data bus (Bus), etc.

[0149] In some embodiments, the processor 102 , the memory 103 , and the I / O interface 104 are connected to each other and further to other components of the computing device via a data bus.

[0150] The above is the complete description of the crosstalk detection device provided by the embodiment of the present disclosure.

[0151] In addition, the embodiment of the present disclosure also provides a holographic communication system. Figure 3 A schematic diagram of a holographic communication system provided by an embodiment of the present disclosure, such as Figure 3 As shown, the holographic communication system includes a plurality of first cameras 104 and a holographic communication device 100 ; wherein the holographic communication device 100 includes a holographic display 101 , a processor 102 and a memory 103 , and the processor 102 and the memory 103 are integrated on a host 105 .

[0152] The plurality of first cameras 104 are configured to capture first images of the user from different perspectives and send the images to the processor 102. The first camera 104 is a color camera (ie, an RGB camera). Optionally, the plurality of first cameras 104 includes, for example, four first cameras 104.

[0153] The memory 103 stores machine-readable instructions executable by the processor 102. When the machine-readable instructions are executed by the processor 102, the processor 102 performs operations, which include the above-mentioned steps S11 to S16. Specifically, based on the received multiple first images, the three-dimensional coordinates of the user's human eyes are determined in real time; the multiple first images are images of the user taken by the multiple first cameras 104 from different perspectives; the video stream is received and decoded into multiple second images; the multiple second images are images of the target object taken by the different second cameras 106 from different perspectives; wherein the target object is The invention relates to a physical object in the operating environment of other devices; the other devices are communication devices that communicate with the holographic communication device 100; the second camera 106 communicates with the other devices; based on the sequence information of each second image, the disparity between the perspectives of two adjacent second images is calculated respectively to obtain multiple disparity maps, and each disparity map is converted into point cloud data; for any disparity map, based on each point cloud data, the image features of the second image are converted to the perspective of the user corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map; the viewpoint feature map is rendered using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's two eyes.

[0154] The holographic display 101 is configured to perform holographic display based on each new viewpoint image.

[0155] Traditional holographic communication systems have shortcomings such as high system complexity, reliance on high-quality depth sensors, low rendering quality, and high computing power requirements. The holographic communication system provided by the embodiments of the present disclosure utilizes a disparity estimation algorithm to solve three-dimensional information and combines it with Gaussian transformation and deferred rendering neural network technology to improve the rendering quality under sparse RGB camera applications, allowing users to obtain better three-dimensional visual effects without wearing peripheral auxiliary equipment. In addition, compared with traditional light field display technology (which requires the simultaneous generation of dozens of new viewpoints), the embodiments of the present disclosure greatly reduce the number of synthesized new viewpoints (only two viewpoints for the user's eyes need to be generated), allowing users to obtain strong three-dimensional visual effects without wearing peripheral auxiliary equipment, enhancing the sense of presence and immersion in remote communication. In addition, the embodiments of the present disclosure only integrate a small number of RGB cameras. This sparse RGB camera configuration greatly reduces the cost of the holographic communication system compared to the more complex image acquisition equipment in traditional technologies (such as dense camera arrays and / or high-precision depth cameras), making it easier to obtain and popularize.

[0156] This disclosure addresses the situation of sparse RGB cameras and limited computing power, and achieves high-quality holographic communication. For example, only four RGB cameras and a consumer-grade graphics card are needed to achieve high-quality communication. Combined with low-latency eye tracking technology, the hardware complexity and cost of such systems are significantly reduced. Tests show that the holographic communication system provided by this disclosure has a communication delay of less than 150ms, a resolution of 2048×2048, and a frame rate of more than 30FPS.

[0157] In addition, the embodiments of the present disclosure further provide a holographic communication method, which is applied to the holographic communication device 100 in any one of the above embodiments or a combination of multiple embodiments. Figure 4 A flow chart of a holographic communication method provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the holographic communication method includes steps S41 to S47.

[0158] S41 . Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images.

[0159] The plurality of first images are images of the user captured by the plurality of first cameras 104 from different perspectives.

[0160] It should be noted that the execution principle of step S41 in the embodiment of the present disclosure is the same as the operation step S11 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0161] S42: Receive a video stream, and decode the video stream into multiple second images.

[0162] Among them, the multiple second images are images of the target object taken by different second cameras 106 from different perspectives; the target object is a physical object in the operating environment of other devices; the other devices are communication devices that communicate with the holographic communication device 100; the second camera 106 communicates with other devices.

[0163] It should be noted that the execution principle of step S42 in the embodiment of the present disclosure is the same as that of step S12 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0164] S43 . Based on the sequence information of each second image, respectively calculate the inter-view disparity of two adjacent second images to obtain a plurality of disparity maps.

[0165] It should be noted that the execution principle of step S43 in the embodiment of the present disclosure is the same as that of step S13 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0166] S44. Convert each disparity map into point cloud data.

[0167] It should be noted that the execution principle of step S44 in the embodiment of the present disclosure is the same as that of step S14 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0168] S45 . For any disparity map, based on each point cloud data, convert the image features of the second image to the user's perspective corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map.

[0169] It should be noted that the execution principle of step S45 in the embodiment of the present disclosure is the same as that of step S15 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0170] S46. Render the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's binocular eyes.

[0171] It should be noted that the execution principle of step S46 in the embodiment of the present disclosure is the same as that of step S16 executed by the processor 102 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0172] S47. Perform holographic display based on the new viewpoint images corresponding to the user's two eyes.

[0173] It should be noted that the execution principle of step S47 in the embodiment of the present disclosure is the same as the operation performed by the holographic display 101 in the above-mentioned holographic communication device 100, and the repeated parts will not be repeated.

[0174] The holographic communication method provided by the disclosed embodiments utilizes a parallax estimation algorithm to calculate three-dimensional information and combines it with Gaussian transformation and deferred rendering neural network technology to improve rendering quality in sparse RGB camera applications, allowing users to obtain better three-dimensional visual effects without wearing peripheral auxiliary equipment. In addition, compared to traditional light field display technology (which requires the simultaneous generation of dozens of new viewpoints), the disclosed embodiments significantly reduce the number of synthesized new viewpoints (only two viewpoints for the user's eyes need to be generated), allowing users to obtain strong three-dimensional visual effects without wearing peripheral auxiliary equipment, thereby enhancing the sense of presence and immersion in remote communication.

[0175] In addition, the embodiment of the present disclosure also provides an interaction method. Figure 5 A schematic diagram of an interactive method provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the interactive method is applied to the different holographic communication systems provided above to form an interactive system. Taking two users (respectively referred to as the first user and the second user) as an example, the interactive system includes a first holographic communication system and a second holographic communication system; wherein the first holographic communication system includes multiple first cameras 104 and a first holographic communication device; and the second holographic communication system includes multiple second cameras 106 and a second holographic communication device. wherein the first holographic communication device and the second holographic communication device are communicatively connected. The multiple first cameras 104 are electrically connected to the first holographic communication device, and the multiple second cameras 106 are electrically connected to the second holographic communication device. The first holographic communication device includes a first holographic display 101-1, a first processor 102-1, and a first memory (not shown in the figure); the second holographic communication device 100 includes a second holographic display 101-2, a second processor 102-2, and a second memory (not shown in the figure).

[0176] The plurality of first cameras 104 are configured to capture first images of a first user from different perspectives and send the first images to the first processor 102 - 1 ; the first user is a user using the first holographic communication device.

[0177] The plurality of second cameras 106 are configured to capture second images of a second user from different perspectives and send the second images to the second processor 102 - 2 ; the second user is a user using the second holographic communication device.

[0178] The second processor 102-2 is configured to encode the received second image of the second user to generate a first video stream and send it to the first processor 102-1. For example, the second processor 102-2 may use WebRTC technology to send the first video stream to the first processor 102-1.

[0179] The first memory stores machine-readable instructions executable by the first processor 102-1. When the machine-readable instructions are executed by the processor 102, the first processor 102-1 performs operations, including: determining the three-dimensional coordinates of the human eyes of the first user in real time based on multiple received first images of the first user; receiving a first video stream and decoding the first video stream into multiple second images; based on the sequence information of each second image, calculating the disparity between perspectives of two adjacent second images respectively to obtain multiple first disparity maps, and converting each first disparity map into first point cloud data; for any first disparity map, based on each first point cloud data, converting the image features of the second image into the three-dimensional coordinates of the human eyes of the first user to obtain a first viewpoint feature map; using a pre-trained rendering neural network model to render the first viewpoint feature map to obtain first new viewpoint maps corresponding to the first user's two eyes respectively.

[0180] The first holographic display 101 - 1 is configured to perform holographic display based on each first new viewpoint image to provide the first user with a stereoscopic holographic image of the second user communicating with the first user.

[0181] At the same time, due to information interaction, the first processor 102 - 1 is also configured to encode the received first image of the first user, generate a second video stream, and send it to the second processor 102 - 2 .

[0182] The second memory stores machine-readable instructions executable by the second processor 102-2. When the machine-readable instructions are executed by the processor 102, the second processor 102-2 performs operations, including: determining the three-dimensional coordinates of the second user's human eyes in real time based on multiple received second images of the second user; receiving a second video stream and decoding the second video stream into multiple first images; based on the sequence information of each first image, calculating the disparity between the perspectives of two adjacent first images respectively to obtain multiple second disparity maps, and converting each second disparity map into second point cloud data; for any second disparity map, based on each second point cloud data, converting the image features of the first image into the three-dimensional coordinates of the second user's human eyes to obtain a second viewpoint feature map; using a pre-trained rendering neural network model to render the second viewpoint feature map to obtain second new viewpoint maps corresponding to the second user's binoculars.

[0183] The second holographic display 101 - 2 is configured to perform holographic display based on each second new viewpoint image to provide the second user with a stereoscopic holographic image of the first user communicating with the second user.

[0184] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0185] In addition, the embodiments of the present disclosure further provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of any of the holographic communication methods described in the above embodiments.

[0186] In particular, based on the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a machine-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0187] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A holographic communication device, wherein: including a holographic display, a processor, and memory; The memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the processor performs operations, the operations including: Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images, wherein the multiple first images are images of the user captured by multiple first cameras from different perspectives; receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device; Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data; For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image into the three-dimensional coordinates of the human eye to obtain a viewpoint feature map; Rendering the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's binocular eyes; The holographic display is configured to perform holographic display based on each of the new viewpoint images.

2. The holographic communication device according to claim 1, wherein: In the operation, based on the sequence information of each second image, the inter-view disparity of two adjacent second images is calculated to obtain multiple disparity maps, including: Based on the sequence information of each second image, a pre-trained disparity estimation model is used to calculate the disparity between the perspectives of two adjacent second images to obtain multiple disparity maps.

3. The holographic communication device according to claim 1, wherein: The operation of converting each disparity map into point cloud data includes: For any of the disparity maps, convert the disparity map into a depth map based on parameter information of the second camera corresponding to the disparity map; Based on the parameter information of the second camera and the position information of the foreground point in the second image corresponding to the depth map, the depth map is converted from the image two-dimensional coordinate system to the human eye three-dimensional coordinate system to obtain a point cloud set; the point cloud set includes point cloud data converted based on one of the depth maps.

4. The holographic communication device according to claim 3, wherein: Determine the depth map using the following formula 1: Formula 1: Among them, f x represents the focal length of the first camera in the X-axis direction of the second camera used to capture the selected second image in the two-dimensional coordinate system of the image; B represents the baseline of the second camera used to capture the selected second image; d represents the grayscale value of a pixel point in the disparity map; and D represents the depth value of the corresponding pixel point in the disparity map.

5. The holographic communication device according to claim 3, wherein: Using the following expression 2, determine a point cloud data in the point cloud set; Formula 2: Where p represents a point cloud data; x, y, z represent the coordinates of the foreground point in the three-dimensional coordinate system of the human eye; x′, y′ represent the coordinates of the foreground point in the two-dimensional coordinate system of the image; f x ,f y They respectively represent the second camera used to capture the selected second image, the first camera focal length in the X-axis direction and the second camera focal length in the Y-axis direction in the two-dimensional coordinate system of the image.

6. The holographic communication device according to claim 1, wherein: The operation converts the image features of the second image into the three-dimensional coordinates of the human eye based on each of the point cloud data to obtain a viewpoint feature map, including: For each of the point cloud data, regressing a Gaussian distribution tensor corresponding to the point cloud data using a convolutional neural network based on image features of the second image; Obtaining Gaussian point cloud data corresponding to the point cloud data based on a tensor corresponding to the Gaussian distribution of the point cloud data; The Gaussian point cloud data are rasterized to the three-dimensional coordinates of the human eye using a Gaussian splattering algorithm to obtain a viewpoint feature map.

7. The holographic communication device according to claim 6, wherein: Use the following formula 3 to determine the Gaussian point cloud data corresponding to the point cloud data; Formula 3: Where Σ=RSS T R T Represents the intermediate result of the matrix operation, R represents the quaternion of the rotation parameter in the tensor, S represents the scale in the tensor, () T Represents the matrix transpose operation; G(p) represents the Gaussian point cloud data corresponding to the point cloud data; p represents the point cloud data.

8. The holographic communication device according to claim 1, wherein: The rendering neural network model includes a UNet convolutional neural network with a residual network structure.

9. The holographic communication device according to claim 2, wherein: The operation further includes training the disparity estimation model, specifically including: Based on sequence information of each sample image in a training data set and the virtual object to which each sample image belongs, selecting a target sample image of the virtual object at a set of adjacent viewing angles from the training data set; the training data set includes sample images of different virtual objects captured by a virtual camera at different viewing angles; Inputting a set of the target sample images into the disparity estimation model to calculate the inter-view disparity and output a theoretical disparity map; A loss function is constructed based on the theoretical disparity map and a preset real disparity map, and the loss function is used for backpropagation to continuously train the disparity estimation model; target sample images of the virtual object under another set of adjacent perspectives are continuously selected from the training data set, or target sample images of other virtual objects under a set of adjacent perspectives are selected from the training data set, and the process of inputting a set of the target sample images into the disparity estimation model to calculate the disparity between perspectives and outputting the theoretical disparity map is returned to execute until the loss function converges to obtain a trained disparity estimation model.

10. The holographic communication device according to claim 9, wherein: The operation further includes training the rendering neural network model, specifically including: Get the pre-rendered theoretical depth map; Based on the parameter information of the virtual camera, the theoretical depth map is converted from the sample image two-dimensional coordinate system to a virtual three-dimensional coordinate system to obtain a training point cloud set; the training point cloud set includes theoretical point cloud data obtained based on the conversion of the theoretical depth map; For each of the theoretical point cloud data, based on the image features of the target sample image, using a convolutional neural network, regressing the tensor corresponding to the Gaussian distribution of the theoretical point cloud data; Obtain theoretical Gaussian point cloud data corresponding to the theoretical point cloud data based on a tensor corresponding to the Gaussian distribution of the theoretical point cloud data; Using a Gaussian splattering algorithm, rasterizing each of the theoretical Gaussian point cloud data to a target viewing angle of the virtual object to obtain a theoretical viewpoint feature map; Rendering the theoretical viewpoint feature map using the rendering neural network model to be trained to obtain theoretical viewpoint maps corresponding to the binocular eyes of the virtual object; A loss function is constructed based on the theoretical viewpoint map and a preset real viewpoint map, and the loss function is used for backpropagation to continuously train the rendering neural network model; and the step of obtaining the pre-rendered theoretical depth map is returned and executed sequentially until the loss function converges to obtain a trained rendering neural network model.

11. A holographic communication system comprising a plurality of first cameras and a holographic communication device; the holographic communication device comprising a holographic display, a processor, and a memory; a plurality of the first cameras, configured to capture first images of the user from different perspectives respectively, and send the first images to the processor; The memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the processor performs operations, the operations including: Determining the three-dimensional coordinates of the user's eyes in real time based on the multiple first images received; The multiple first images are images of the user captured by multiple first cameras from different perspectives; receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device; Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data; For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image to the user's perspective corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map; Rendering the viewpoint feature map using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's binocular eyes; The holographic display is configured to perform holographic display based on each of the new viewpoint images.

12. The holographic communication system according to claim 11, wherein: The first camera is a color camera.

13. The holographic communication system according to claim 11 or 12, wherein: The plurality of first cameras includes four first cameras.

14. A holographic communication method, applied to the holographic communication device according to any one of claims 1 to 10; wherein: The holographic communication method comprises: Determine the three-dimensional coordinates of the user's eyes in real time based on the received multiple first images, wherein the multiple first images are images of the user captured by multiple first cameras from different perspectives; receiving a video stream and decoding the video stream into a plurality of second images; the plurality of second images are images of a target object captured by different second cameras from different perspectives; wherein the target object is a physical object in an operating environment of another device; the other device is a communication device that communicates with the holographic communication device; and the second camera communicates with the other device; Based on the sequence information of each second image, respectively calculating the disparity between the perspectives of two adjacent second images to obtain a plurality of disparity maps, and converting each of the disparity maps into point cloud data; For any of the disparity maps, based on each of the point cloud data, convert the image features of the second image to the user's perspective corresponding to the three-dimensional coordinates of the human eye to obtain a viewpoint feature map; The viewpoint feature map is rendered using a pre-trained rendering neural network model to obtain new viewpoint maps corresponding to the user's two eyes, and then holographically displayed.

15. A computer non-transitory readable storage medium, wherein: The computer non-transitory readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the holographic communication method as claimed in claim 14 .

Images