Holographic communication method, electronic device, storage medium, and program product

Abstract

This document discloses a holographic communication method, electronic device, storage medium, and program product. The method includes: acquiring a three-dimensional model of a first object; the three-dimensional model is obtained based on multiple depth images and multiple attribute images, the multiple depth images and multiple attribute images being obtained based on multiple first images, the multiple first images being acquired from multiple viewpoints of the first object, the multiple attribute images including attribute data of multiple primitives under the multiple viewpoints, the primitives being basic units representing the first object; and rendering and displaying the three-dimensional model based on the three-dimensional model and the client's viewpoint. Therefore, high-fidelity stereoscopic image reconstruction can be achieved on the client using the acquired multi-view images, while ensuring viewpoint integrity.

Description

Technical Field

[0001] This document relates to the field of communication technology, and in particular to a holographic communication method, electronic device, storage medium, and program product. Background Technology

[0002] Holographic communication, as a novel interactive method integrating holographic imaging and communication technologies, focuses on the acquisition and transmission of three-dimensional holographic data of objects, and the reconstruction of the object's stereoscopic image at the receiving end. This technology supports real-time, immersive, and remote interaction and is currently widely used in key scenarios such as remote conferencing, medical consultations, and industrial collaboration.

[0003] The core foundation for achieving high-fidelity stereoscopic image reconstruction at the receiving end using 3D holographic data while ensuring viewpoint integrity is still needed. An effective solution for this is required. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a holographic communication method, electronic device, storage medium, and program product to solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the embodiments in this specification adopt the following technical solutions: Firstly, a holographic communication method is provided for application on a client side, including: A three-dimensional model of a first object is obtained; the three-dimensional model is obtained based on multiple depth images and multiple attribute images, the multiple depth images and multiple attribute images are obtained based on multiple first images, the multiple first images are acquired from multiple perspectives of the first object, the multiple attribute images include attribute data of multiple primitives under the multiple perspectives, and the primitives are basic units representing the first object; The 3D model is rendered and displayed based on the client's perspective.

[0006] Secondly, a holographic communication method is provided for application on the server side, including: Receive a second data stream, the second data stream including a plurality of first images, the plurality of first images being acquired from multiple perspectives of a first object; Depth estimation is performed on the plurality of first images to obtain a plurality of depth images; Based on the plurality of first images and the plurality of depth images, a plurality of attribute images are obtained. The plurality of attribute images include attribute data of a plurality of primitives under the plurality of viewpoints. The primitives are basic units representing the first object. Based on the plurality of depth images and the plurality of attribute images, a first data stream is obtained and sent to the client. The first data stream is used to render and display the three-dimensional model of the first object.

[0007] Thirdly, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the holographic communication method provided in the first aspect or the holographic communication method provided in the second aspect.

[0008] Fourthly, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a holographic communication method as provided in the first aspect or a holographic communication method as provided in the second aspect.

[0009] Fifthly, a computer program product is provided, the computer program product including a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in the holographic communication method provided in the first aspect, or the computer program operable to cause a computer to perform some or all of the steps in the holographic communication method provided in the second aspect.

[0010] The scheme of this embodiment acquires multiple first images of a first object from different perspectives, and these first images contain multi-view information of the first object. Multiple depth images are obtained by performing depth estimation on these first images. Based on the depth images, 3D reconstruction technology is combined to perform depth aggregation and collaborative processing on the multi-view information to obtain multiple attribute images. These attribute images reflect the attribute data of multiple primitives (i.e., the basic units representing the first object) from multiple perspectives, providing sufficient data support for the 3D reconstruction of the first object. Based on these attribute images and depth images, the 3D spatial information of the first object can be completely captured, the detailed features of the first object can be accurately restored, and problems such as information loss and perspective fragmentation caused by single-view acquisition can be effectively avoided. This allows for high-fidelity stereoscopic image restoration on the client side while ensuring perspective integrity, enhancing the immersive and realistic experience of the user's holographic interaction. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of an example environment in which the embodiments of this specification can be implemented; Figure 2 One of the flowcharts of a holographic communication method provided in one embodiment; Figure 3 A second schematic flowchart of a holographic communication method provided for one embodiment; Figure 4 A third schematic flowchart of a holographic communication method provided for one embodiment; Figure 5 A fourth schematic flowchart illustrating a holographic communication method provided as an embodiment; Figure 6 A schematic diagram of image acquisition for a first object is provided as an embodiment; Figure 7 Fifth of a flowchart illustrating a holographic communication method provided in one embodiment; Figure 8 A flowchart illustrating a holographic communication method as provided in one embodiment is shown in Figure 6. Figure 9 A flowchart illustrating an image compression method provided in one embodiment; Figure 10 This is a schematic diagram of the structure of an electronic device provided in one embodiment. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this document.

[0013] The term "comprising" and its variations as used in this document are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. The term "in response to" indicates that the performed operation depends on a condition or state. When the dependent condition or state is met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which multiple operations are performed.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this document are used only to distinguish different devices, modules or units, and are not used to restrict the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "one" and "more" used in this document are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in this document are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] In this document, any processing of personal information will be conducted only on a legal basis (such as with the consent of the data subject or as necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to have personal information beyond what is necessary for basic functions processed will not affect their use of basic functions.

[0018] As mentioned earlier, the core foundation for achieving high-fidelity stereoscopic image reconstruction at the receiving end using 3D holographic data while ensuring viewpoint integrity is crucial for realizing high-quality holographic displays. Currently, the processing of 3D holographic data often involves simply superimposing images from different viewpoints, resulting in incomplete aggregation of multi-view images. This leads to computational redundancy in the subsequent stereoscopic image reconstruction stage and directly affects the display clarity and viewpoint integrity of the holographic image.

[0019] In view of this, this specification proposes a holographic communication method. Multiple first images are obtained by acquiring images of a first object from different perspectives, and these images are transmitted to a server. These first images contain multi-view information about the first object. Multiple depth images are obtained by performing depth estimation on these first images. Based on these depth images, 3D reconstruction technology is combined to perform depth aggregation and collaborative processing on the multi-view information, resulting in multiple attribute images. These attribute images reflect the attribute data of multiple primitives (i.e., the basic units representing the first object) from multiple perspectives, providing sufficient data support for the 3D reconstruction of the first object. Based on these attribute images and depth images, the 3D spatial information of the first object can be completely captured, accurately restoring the detailed features of the first object. This effectively avoids problems such as information loss and perspective fragmentation caused by single-view acquisition, thereby achieving high-fidelity stereoscopic image restoration at the receiving end while ensuring perspective integrity and enhancing the immersive and realistic experience of the user's holographic interaction.

[0020] Optionally, considering the shortcomings of traditional mesh-based 3D reconstruction techniques in terms of surface detail representation and lighting accuracy, 3DGS (3D Gaussian Splatting) technology is used to achieve 3D reconstruction of the first object. 3DGS technology uses a set of Gaussian primitives as graphic elements. These primitives possess attribute data such as position, rotation, opacity, scale, and spherical harmonic coefficients (i.e., color). Based on these primitive attribute data, 3D reconstruction and rendering are performed at the receiving end, achieving accurate conversion from multi-view images to 3D scenes. This ensures the realism and fine-grainedness of the rendered stereoscopic image at the receiving end, while also considering the quality and efficiency of rendering new perspectives after reconstruction, thus better meeting the application requirements of high-definition holographic communication.

[0021] Optionally, considering that the 3D reconstruction process typically consumes significant computing resources and is time-consuming, it is not conducive to the real-time conversion of multi-view images to stereoscopic images, making it difficult to meet the core requirement of holographic communication for "real-time interaction." Therefore, on the one hand, the process of generating multiple attribute images on the server side is optimized to quickly complete the conversion of multi-view images to attribute images, improving reconstruction efficiency; on the other hand, the rendering process on the receiving end is optimized, relying on high-performance rendering technology to achieve real-time rendering, significantly reducing the latency of holographic image display. These two collaborative optimizations achieve low-latency linkage throughout the entire process of "acquisition-reconstruction-transmission-rendering-display," ensuring the real-time nature of holographic communication and thus meeting the usage requirements in scenarios such as remote interaction and real-time display.

[0022] Optionally, considering the massive amount of primitives and their attribute data generated during 3D reconstruction, if traditional single-dimensional dimensionality reduction optimization techniques are used to compress this data, a collaborative balance mechanism between rendering quality, transmission bandwidth, and real-time performance cannot be established. This can easily lead to the dilemma of "over-compression causing rendering distortion" or "lightweight compression failing to adapt to low-bandwidth transmission," hindering the large-scale application of 3D reconstruction technology in holographic communication scenarios. Therefore, the holographic communication method provided in the embodiments of this specification also designs corresponding lightweight compression strategies for depth images and attribute images, respectively. This can significantly reduce the amount of data transmitted while ensuring that the reconstruction quality is not affected, effectively reducing transmission bandwidth usage, while simultaneously increasing the data transmission rate. This avoids problems such as transmission stuttering and packet loss due to insufficient bandwidth, achieving stable and efficient transmission of holographic data and broadening the application scenarios of holographic communication technology (such as remote holographic communication in low-bandwidth scenarios).

[0023] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram illustrates an example environment in which embodiments of this specification can be implemented. For example... Figure 1As shown, this example environment may include a data acquisition terminal, a receiving terminal, and a server. Communication connections are established between the data acquisition terminal and the server, and between the receiving terminal and the server.

[0025] The acquisition terminal has a built-in acquisition module that can acquire multi-view video of a first object in real time and send the multi-view video to the server. The multi-view video may include audio signals and multi-view images. In some embodiments, the acquisition terminal deploys a client that provides acquisition functionality; the client may include the acquisition module. The acquisition terminal may include various terminals, such as holographic displays. For example, multiple cameras are arranged around the perimeter of a holographic display device; the acquisition module can trigger these cameras to acquire images of an object from multiple perspectives, obtaining multi-view images of that object. The object here may include, but is not limited to, the environment, users, and other objects.

[0026] The server-side includes a reconstruction module and a transmission module. The reconstruction module acquires multi-view images from multi-view video, performs depth estimation on these images to obtain depth images for each view, and then uses 3D reconstruction technology to perform depth aggregation and co-processing on these depth images to obtain attribute images for each view. Further, the transmission module obtains a data stream based on the audio signal, depth images from all views, and attribute images from all views, and sends the data stream to the receiving end. For example, the transmission module can package the audio signal, depth images, and attribute images into a data stream, or it can reconstruct a 3D model of the object based on the depth images and attribute images, and then compress the 3D model and audio signal to obtain the data stream. In some embodiments, the server-side may include a server. The server can be a cloud server, a standalone physical server, or a server cluster or distributed system consisting of multiple physical servers.

[0027] The receiving end has a built-in rendering module. Based on the server, the rendering module obtains the 3D model of the aforementioned object and renders and displays the 3D model based on the receiving end's perspective. The rendering module can also play audio signals. In some embodiments, a client with display capabilities is deployed in the receiving end; the client may include the rendering module. The receiving end may include various terminals, such as holographic displays. The receiving end's perspective can be understood as the user's perspective. This perspective may include the user's left-eye and right-eye perspectives. The user's perspective can be detected in real-time by the receiving end, for example, through sensors on the receiving end.

[0028] Figure 2 This is a flowchart illustrating a holographic communication method according to one embodiment. The method can be applied to a client, which may include a client deployed on the receiving end. The method may include the following steps: S202, Obtain the three-dimensional model of the first object. The three-dimensional model is obtained based on multiple depth images and multiple attribute images.

[0029] Multiple depth images and multiple attribute images are obtained based on multiple first images. These multiple first images are acquired from multiple viewpoints of the first object. In some embodiments, the first images may be color images (i.e., RGB images). In some embodiments, the multiple first images may be acquired by an acquisition device.

[0030] Multiple depth images, multiple attribute images, multiple first images, and multiple viewpoints correspond to each other. The first object can be any object that needs to be transmitted to the client for holographic display, such as, but not limited to, the environment, the user, and other objects.

[0031] Multiple attribute images include attribute data of multiple primitives from multiple viewpoints; that is, each attribute image includes attribute data of multiple primitives from the viewpoint corresponding to that attribute image. For each first image, each pixel contains a corresponding primitive.

[0032] Primitives are the basic units representing a primary object. They are the smallest geometric shape units that can be directly identified and processed during the rendering process. These geometric shapes can include, but are not limited to, ellipsoids, spheres, and other geometric states composed of one or more vertices combined according to specific rules. Primitive attribute data can be understood as data used to describe the attributes of the primitive, which can include, but are not limited to, the primitive's color, position, opacity, and scale.

[0033] Considering the shortcomings of traditional mesh-based 3D reconstruction techniques in terms of surface detail representation and lighting accuracy, 3DGS technology is used in some embodiments to achieve 3D reconstruction of the first object. In this case, primitives may include Gaussian primitives, and the attribute data of the primitives may include, but is not limited to, at least one of the following: opacity, scale, shape, position, and color of the Gaussian primitive. A Gaussian primitive is a semi-transparent, colored "Gaussian sphere" or "ellipsoid" in 3D space. These Gaussian primitives collectively describe the geometry, appearance, and other attributes of the first object. The shape of the Gaussian primitive is determined by its rotation and scaling matrices, which allows the Gaussian primitive to be stretched, rotated, and transformed into an ellipsoid to better fit the surface of the first object. The position of the Gaussian primitive may include the center coordinates of the Gaussian primitive in 3D space, and the color of the Gaussian primitive may include SH (Spherical Harmonics) coefficients.

[0034] In some embodiments, the three-dimensional model of the first object can be obtained by processing the received first data stream.

[0035] In some cases, the server can obtain and send the first data stream to the client based on multiple first images. The process by which the server obtains the first data stream will be described in detail later.

[0036] Accordingly, the above S202 may include: receiving a first data stream, the first data stream including multiple depth images and multiple attribute images, and back-projecting the multiple depth images and multiple attribute images into a three-dimensional space to obtain a three-dimensional model.

[0037] By backprojecting multiple depth images and multiple attribute images into 3D space, the precise matching of the 3D coordinates and attribute parameters of multiple primitives is achieved. Ultimately, these primitives are recombined into a 3D model of the first object, providing data support for subsequent rendering. In other words, a 3D model can include multiple primitives. For the backprojection of attribute images, existing techniques can be referenced, and will not be elaborated further.

[0038] In other cases, S202 may include: receiving a first data stream, wherein the first data stream is obtained by the server reconstructing a three-dimensional model of a first object based on multiple depth images and multiple attribute images, and then compressing the three-dimensional model; and decompressing the first data stream to obtain the three-dimensional model.

[0039] In some other cases, the above S202 may include: receiving a first data stream, which may be obtained by encoding a second image, the second image being obtained by stitching together multiple attribute images and multiple depth images; decoding the first data stream to obtain the second image, and separating multiple attribute images and multiple depth images from the second image; and back-projecting the multiple attribute images and multiple depth images into a three-dimensional space to obtain a three-dimensional model.

[0040] In other cases, the above S202 may include: receiving a first data stream, which may include a plurality of first images; performing depth estimation on the plurality of first images to obtain a plurality of depth images; obtaining a plurality of attribute images based on the plurality of first images and the plurality of depth images; and back-projecting the plurality of attribute images and the plurality of depth images into a three-dimensional space to obtain a three-dimensional model.

[0041] The above illustrates a partial implementation of S202. It should be understood that S202 can also be implemented in other ways, and no limitation is made thereto.

[0042] S204 renders and displays 3D models based on the 3D model and the client's perspective.

[0043] The perspective here can be understood as the user's perspective on the client side. This perspective can include the user's left-eye and right-eye perspectives. In some implementations, such as... Figure 3As shown, the user's left and right eye poses can be detected in real time by the sensors at the receiving end (such as a holographic display) where the client is located, and then the left eye pose is used as the left eye view and the right eye pose is used as the right eye view.

[0044] In other implementations, the client's perspective can also be a perspective specified by the client's user. For example, the client can present multiple perspective options for the user to choose from, and in response to the user's selection of a perspective option, the client will use the perspective corresponding to the selected perspective option as the client's perspective. Alternatively, the client may present...

[0045] In S204 above, the rendering and display of the 3D model can be achieved in various ways.

[0046] In some implementations, the rendering mechanism of 3DGS reconstruction technology can be used to render and display the 3D model. Specifically, S204 above may include the following steps: S2042, based on the client's perspective, projects multiple primitives onto the client's two-dimensional imaging plane to obtain the number of pixels covered by each primitive on the two-dimensional imaging plane and the depth value of each primitive.

[0047] The two-dimensional imaging plane can include the imaging plane of the receiving end (such as a holographic display) where the client is located. By projecting multiple primitives onto the two-dimensional imaging plane, these primitives are transformed from three-dimensional to two-dimensional, completing the mapping from the three-dimensional spatial information of the three-dimensional model to the two-dimensional imaging plane. For each primitive, the pixels covered by the projection of that primitive onto the two-dimensional imaging plane are the pixels covered by that primitive on the two-dimensional imaging plane.

[0048] Since the attribute data of a primitive includes its position, which is the primitive's coordinates in world coordinates, the primitive's position is transformed from the world coordinate system to the camera coordinate system according to the client's perspective, thus obtaining the primitive's coordinates in the camera coordinate system. Furthermore, the z-component of the primitive in the camera coordinate system is extracted to obtain the primitive's depth value.

[0049] S2044, for each pixel on the two-dimensional imaging plane, the color of at least one first primitive is mixed based on the depth value of at least one first primitive covering the pixel to obtain a rendered image from the client's perspective.

[0050] In some cases, for each pixel on the two-dimensional imaging plane, these first primitives can be sorted based on the depth value of at least one first primitive covering the pixel, such as in ascending or descending order. Then, the colors of these first primitives are superimposed according to their sorting order to obtain the color of the pixel. After color mixing of all pixels is completed, the rendered image from the client's perspective is obtained. The rendered image includes a pixel array that meets the resolution requirements of holographic displays.

[0051] In this process, for the superposition of the colors of at least the first primitive, the alpha blending technique can be used to determine the contribution of each first primitive to the pixel based on the opacity of each first primitive. Furthermore, the colors of at least one first primitive are weighted and superimposed based on the contribution of each first primitive to the pixel to obtain the color of the pixel.

[0052] In other cases, before S2044, the process may further include: dividing the two-dimensional imaging plane into multiple regions; for each region, sorting the primitives covering the region based on their depth values ​​to obtain an arrangement order of the primitives within that region. Accordingly, S2044 may include: for each pixel on the two-dimensional imaging plane, superimposing the color of at least one first primitive onto that pixel based on its arrangement order within the region. This yields the color of that pixel. After color mixing of all pixels is completed, a rendered image from the client's perspective is obtained.

[0053] The above method enables block-based parallel rendering of the two-dimensional imaging plane. Furthermore, sorting the primitives covering each region according to their depth values ​​ensures the correct handling of occlusion relationships during rendering and avoids imaging distortion.

[0054] S2046, Display the rendered image.

[0055] It is worth noting that since the client's perspective includes both left-eye and right-eye views, the rendered image from this perspective can include both the rendered image from the left-eye view and the rendered image from the right-eye view. By displaying the rendered image from the left-eye view and the rendered image from the right-eye view, the stereoscopic image of the first object is restored.

[0056] The above implementation method enables rapid 3D reconstruction based on the first data stream and real-time rendering adapted to the client's perspective, significantly reducing the latency of holographic image display.

[0057] like Figure 3As shown, the client can continuously receive and process the first data stream. During processing, the client detects the user's left and right eye poses in real time to obtain the client's viewpoint; furthermore, the 3D model is reconstructed, rendered, and displayed through the above steps S202~S204. Thus, real-time holographic communication is achieved.

[0058] The foregoing illustrates a partial implementation of S204. It should be understood that S204 can also be implemented in other ways, such as using a mesh-based 3D reconstruction rendering mechanism to render and display the 3D model, etc. The embodiments in this specification do not limit this to such methods.

[0059] The holographic communication method provided in the above embodiments acquires multiple first images of a first object from different perspectives. These first images contain multi-view information about the first object. Multiple depth images are obtained by depth estimation of these first images. Based on these depth images, 3D reconstruction technology is combined to perform depth aggregation and collaborative processing on the multi-view information, resulting in multiple attribute images. These attribute images reflect the attribute data of multiple primitives (i.e., the basic units representing the first object) from multiple perspectives, providing sufficient data support for the 3D reconstruction of the first object. Based on these attribute images and depth images, the 3D spatial information of the first object can be completely captured, accurately restoring the detailed features of the first object. This effectively avoids problems such as information loss and perspective fragmentation caused by single-view acquisition, thereby achieving high-fidelity stereoscopic image restoration on the client side while ensuring perspective integrity and enhancing the immersive and realistic experience of the user's holographic interaction.

[0060] Figure 4 The following is a flowchart illustrating a holographic communication method provided in one embodiment. This method can be applied to a server and may include the following steps: S402, Receive a second data stream, the second data stream including a plurality of first images.

[0061] Multiple first images are obtained by capturing the first object from multiple perspectives. Multiple first images can correspond to multiple perspectives.

[0062] In some embodiments, in the second data stream, multiple first images are stitched together to form a panoramic image according to a preset stitching rule. In this case, after receiving the second data stream, the server can split the panoramic image according to the above stitching rule to obtain multiple first images.

[0063] S404, perform depth estimation on multiple first images to obtain multiple depth images.

[0064] Multiple depth images correspond to multiple first images. A depth image includes the depth value of each pixel in the corresponding first image. When the first object is captured from different viewpoints, by calculating the positional offset (i.e., parallax) of feature points in different first images, the depth value of each pixel in the first image can be accurately calculated using geometric principles (i.e., triangulation principles), thus obtaining the depth image corresponding to the first image.

[0065] In one implementation, S404 may include: extracting features from multiple first images to obtain multiple feature images; the multiple feature images correspond to the multiple first images. Each feature image may include image features of each pixel in the corresponding first image; matching the multiple feature images to obtain pixel correspondences among the multiple first images; obtaining the disparity of each pixel in each first image based on the pixel correspondences; and converting the disparity of the pixels into depth values ​​to obtain a depth image corresponding to each first image.

[0066] Specifically, for each first image, feature matching can be performed on the epipolar lines of other first images based on the camera parameters (such as intrinsic and extrinsic parameters) of the first image and the principle of epipolar geometry to obtain the pixel correspondence between the first image and other first images, thereby inferring the depth image corresponding to each first image.

[0067] More specifically, for each feature image, for each pixel in the feature image, its image feature vL is extracted, and on the epipolar lines of other feature images, a series of pixel image features vR1, vR2, etc. are extracted; the distance between image feature vL and each image feature vR1, vR2, etc. is calculated; based on these distances, the disparity of each pixel in the first image corresponding to the feature image is determined; finally, based on the principle of triangulation, using the camera parameters corresponding to the first image, the disparity of each pixel is converted into the depth value of each pixel.

[0068] By performing depth estimation in the above manner, multiple first images can complement each other to supplement the occluded information, thereby improving accuracy.

[0069] It is worth noting that the camera parameters for each first image can be obtained through pre-calibration or predicted by a model (such as a neural network).

[0070] In another implementation, S404 may include: inputting multiple first images into a second model to obtain multiple depth images. The second model can be trained based on sample images from different viewpoints and scenes, and reference depth images corresponding to the sample images. The second model may include a model with image processing capabilities, such as a neural network.

[0071] During training, the reference depth image can be used as a supervision signal to supervise the depth estimation accuracy of the second model on the sample images. For example, the loss of the second model (such as L1 loss) can be calculated based on the error between the depth image output by the second model and the reference depth image, and then the parameters of the second model can be adjusted based on the loss.

[0072] The foregoing illustrates a partial implementation of S404. It should be understood that S404 can also be implemented in other ways, and the embodiments in this specification do not limit this implementation.

[0073] Optionally, before the above S404, it may also include: for each first image, performing foreground and background separation on the first image, and masking the background of the first image.

[0074] Foreground and background separation of the first image can refer to existing techniques, such as background matting algorithms, and will not be elaborated further. Masking the background of the first image can also refer to existing masking techniques, such as setting the pixels representing the background to 0, and will not be elaborated further.

[0075] The first image after masking is the same size as the first image before masking, retaining only the foreground of the first image before masking. Understandably, performing depth estimation on these masked first images not only reduces computational load and significantly improves computational efficiency, but also eliminates background interference on depth estimation, thus improving accuracy.

[0076] S406, based on multiple first images and multiple depth images, obtain multiple attribute images.

[0077] A multi-attribute image includes attribute data of multiple primitives from multiple viewpoints. A primitive is a basic unit representing a first object. In some embodiments, a primitive may include Hexagonal primitives, and the attribute data of the primitives may include, but is not limited to, at least one of the following: opacity, scale, shape, position, color, etc. of the Hexagonal primitives.

[0078] In one implementation, S406 may include: for each first image, inputting the first image and its corresponding depth image into a third model to obtain an attribute image corresponding to the first image. The third model may be trained based on sample images from different viewpoints and scenes, and reference attribute images corresponding to the sample images. The third model may include a model with image processing capabilities, such as a neural network.

[0079] During training, the reference attribute image can be used as a supervision signal to monitor the accuracy of the third model's attribute prediction of the sample image. For example, the loss of the third model can be calculated based on the error between the attribute image output by the third model and the reference attribute image, and then the parameters of the third model can be adjusted based on the loss.

[0080] In another implementation, S406 may include: aligning multiple first images and their corresponding depth images into a unified three-dimensional space to construct a global three-dimensional model, and then using this three-dimensional model to render attribute images from multiple perspectives.

[0081] The foregoing illustrates a partial implementation of S406. It should be understood that S406 can also be implemented in other ways, and the embodiments in this specification do not limit this implementation.

[0082] S408, based on multiple depth images and multiple attribute images, obtains a first data stream and sends the first data stream to the client, the first data stream being used by the client to render and display a 3D model of a first object.

[0083] The server can directly send the first data stream to the client. Alternatively, the server can upload the first data stream to a CDN (Content Delivery Network), and the CDN's node distribution capabilities will then send the first data stream to the client based on the client's access request, ensuring efficient and stable transmission.

[0084] The first data stream can be obtained in various ways, and the embodiments in this specification do not limit this.

[0085] In one implementation, the first data stream includes multiple depth images and multiple attribute images.

[0086] In another implementation, the first data stream is obtained by backprojecting multiple depth images and multiple attribute images into a three-dimensional space to obtain a three-dimensional model of the first object, and then compressing the three-dimensional model.

[0087] In another implementation, the first data stream is generated based on multiple depth images and multiple attribute images after masking. The multiple attribute images and multiple depth images correspond to multiple first images. The multiple attribute images after masking are obtained as follows: for each first image, the first image and its corresponding depth image are input into a first model to obtain a redundant region of the first image; the attribute data of the first primitives in the attribute images corresponding to the first image are masked to obtain a masked attribute image, where the first primitives correspond to pixels in the redundant region.

[0088] The first model can be trained based on sample images from different perspectives and scenes, as well as reference non-redundant regions of the sample images. The first model can include models with image processing capabilities, such as neural networks.

[0089] During training, the reference non-redundant region can serve as a supervisory signal to monitor the accuracy of the first model in recognizing non-redundant regions of the sample image. For example, the loss of the first model can be calculated based on the error between the non-redundant region recognized by the first model and the reference non-redundant region, and then the parameters of the first model can be adjusted based on this loss. In this case, by inputting the first image and its corresponding depth image into the first model, the non-redundant regions of the first image can be obtained; the regions in the first image other than the non-redundant regions are the redundant regions of the first image.

[0090] It is understandable that by masking multiple attribute images in the above way, redundant primitives in each attribute image can be filtered out, the influence of background noise can be eliminated, and computing resources and rendering attention can be fully focused on effective primitives. This not only helps to improve the accuracy and purity of subsequent 3D reconstruction of the first object, but also helps to improve the efficiency of 3D reconstruction and rendering, and better meet the core requirement of holographic communication for "real-time interaction".

[0091] In another implementation, the first data stream is obtained as follows: Step A1: Quantize the multiple attribute images based on the first precision.

[0092] The first precision is less than the precision of multiple attribute images. Quantization of an attribute image can be understood as quantizing the attribute data of each primitive in the attribute image to the first precision.

[0093] For example, if the precision of the attribute image is 32-bit floating-point, the first precision can include 8-bit integers, thereby quantizing the attribute data of each primitive in the attribute image into 8-bit integers, so that these attribute data are mapped to the interval [0,255].

[0094] Step A2: Quantize multiple depth images based on the second precision.

[0095] The second precision is less than the precision of a multi-attribute image. Quantization of a depth image can be understood as quantizing the depth value of each pixel in the depth image to the second precision.

[0096] For example, if the precision of a depth image is a 32-bit floating-point number, then the second precision can include a 12-bit integer, thereby quantizing the depth value of each pixel in the depth image into a 12-bit integer, so that these depth values ​​are mapped to the corresponding numerical range.

[0097] Step A3: The quantized multiple attribute images and the quantized multiple depth images are stitched together to obtain the second image.

[0098] In some cases, these attribute images and depth images can be stitched together according to preset layout rules to obtain a second image, thereby achieving unified encapsulation of multi-dimensional data. The layout rules can be set according to actual needs, such as first stitching multiple attribute images sequentially, and then stitching multiple depth images sequentially after the last attribute image, etc. This specification does not limit this practice in the embodiments.

[0099] In other cases, multiple third images and multiple fourth images can be generated. The multiple third images correspond to multiple quantized attribute images, and the multiple fourth images correspond to multiple quantized depth images. For each quantized attribute image, the attribute data of each primitive in that attribute image is written into the first channel of the corresponding third image to obtain the fifth image. For each quantized depth image, the depth value of each pixel in that depth image is written into the second and third channels of the corresponding fourth image to obtain the sixth image. The fifth and sixth images are then stitched together to obtain the second image. This allows for unified storage of the quantized attribute images and quantized depth images, ensuring the complete preservation of attribute data and depth values.

[0100] The third and fourth images can be all-zero matrices or placeholder images without loaded valid data. The first channel can be determined based on a first precision, and the second and third channels can be determined based on a second precision.

[0101] For example, if the first precision includes 8-bit integers, then the first channel may include the Y channel. It is understood that when an attribute image is quantized to 8-bit integers, the attribute data in the quantized attribute image lies within the range [0, 255], which can significantly reduce the data size of the attribute image. Furthermore, since the Y channel is used to represent grayscale values, and the range of grayscale values ​​is [0, 255], it closely matches the range of attribute data in the quantized attribute image.

[0102] The second precision includes 12-bit integers, so the second channel can include the Y channel, and the third channel can include the U channel. It's understandable that a 12-bit integer ranges from 0 to 4095, representing 4096 levels of depth values. This is a cost-effective precision for most consumer and industrial depth cameras, capable of distinguishing subtle depth changes without consuming too much space. Since a single channel of the fourth image cannot fully store 4096 levels of depth values, the Y and U channels are used to store the quantized depth image. For example, the Y channel stores the high-order bits of the quantized depth values, and the U channel stores the low-order bits, ensuring complete preservation of the depth values.

[0103] Step A4: Encode the second image to obtain the first data stream.

[0104] The second image can be encoded using a high-efficiency video encoder (such as H.265 / VP9) to obtain a first data stream with low bandwidth usage, further reducing the amount of data transmitted.

[0105] Using the above methods, a corresponding lightweight compression strategy was designed for multiple depth images and multiple attribute images. This strategy can significantly reduce the amount of data transmission while ensuring that the reconstruction quality is not affected, effectively reduce the transmission bandwidth usage, and improve the data transmission rate. It avoids problems such as transmission stuttering and packet loss caused by insufficient bandwidth, thereby achieving stable and efficient transmission of holographic data and expanding the application scenarios of holographic communication technology (such as remote holographic communication in low-bandwidth scenarios).

[0106] The holographic communication method provided in the above embodiments acquires multiple first images of a first object from different perspectives and transmits these first images to the server. These first images contain multi-view information of the first object. The server performs depth estimation on these first images to obtain multiple depth images. Based on the depth images, it combines 3D reconstruction technology to perform depth aggregation and collaborative processing on the multi-view information to obtain multiple attribute images. These attribute images reflect the attribute data of multiple primitives (i.e., the basic units representing the first object) from multiple perspectives, providing sufficient data support for the 3D reconstruction of the first object. The server obtains a first data stream based on these attribute images and depth images and sends the first data stream to the client. The client can completely capture the 3D spatial information of the first object, accurately restore the detailed features of the first object, and effectively avoid problems such as information loss and perspective fragmentation caused by single-view acquisition. Thus, high-fidelity stereoscopic image restoration is achieved on the client while ensuring the integrity of the perspective, enhancing the immersiveness and realism of the user's holographic interaction.

[0107] Please refer to Figure 5 The following is a flowchart illustrating a holographic communication method according to another embodiment. This method can be applied to a client, which may include a client deployed at the acquisition end. The method may include the following steps: S502, acquire multiple first images, which are obtained by capturing the first object from multiple perspectives.

[0108] The first object can be captured from multiple perspectives using an image acquisition device (such as a holographic display). The image acquisition device can include multiple cameras, each with a different pose. For example, ... Figure 6As shown, the acquisition end includes a holographic display, and the image acquisition device may include multiple cameras arranged around the perimeter of the holographic display to achieve omnidirectional, blind-spot-free image acquisition of the first object, obtaining multiple first images. These multiple first images correspond to multiple viewpoints.

[0109] It's worth noting that these cameras can employ a hardware synchronization triggering mechanism to ensure strict consistency in the acquisition timing across all viewpoints. For example, a high-precision external trigger source (such as a dedicated synchronization controller) can be used. This source generates a precise electrical signal (such as a TTL pulse) as the synchronization start command for all cameras. This signal is transmitted to each camera via dedicated hardware lines (such as coaxial cable, fiber optic cable, or high-speed bus), avoiding random delays that may be introduced by software communication. The built-in hardware circuitry of each camera directly interprets the electrical signal and initiates image acquisition, without needing processing by the operating system or driver layer, further eliminating software latency.

[0110] Optionally, after the acquisition is completed, the client can stitch multiple first images into a panoramic image according to certain rules, and send the second data stream containing the panoramic image to the server to provide complete and synchronous raw data support for subsequent 3D reconstruction and rendering.

[0111] S504, send a second data stream to the server, the second data stream including multiple first images.

[0112] Multiple first images are used to obtain multiple depth images and multiple attribute images, which are then used by the client, acting as the receiving end, to render and display a 3D model of the first object. The multiple attribute images include attribute data of multiple primitives from multiple viewpoints; each primitive is a basic unit representing the first object.

[0113] Optionally, the client can also acquire the audio signal of the first object. Correspondingly, the second data stream can also include the audio signal.

[0114] The holographic communication method provided in the above embodiments acquires multiple first images of a first object from different perspectives. These first images contain multi-view information about the first object. Multiple depth images are obtained by depth estimation of these first images. Based on these depth images, 3D reconstruction technology is combined to perform depth aggregation and collaborative processing on the multi-view information, resulting in multiple attribute images. These attribute images reflect the attribute data of multiple primitives (i.e., the basic units representing the first object) from multiple perspectives, providing sufficient data support for the 3D reconstruction of the first object. Based on these attribute images and depth images, the 3D spatial information of the first object can be completely captured, accurately restoring the detailed features of the first object. This effectively avoids problems such as information loss and perspective fragmentation caused by single-view acquisition, thereby achieving high-fidelity stereoscopic image restoration on the client side while ensuring perspective integrity and enhancing the immersive and realistic experience of the user's holographic interaction.

[0115] To facilitate a better understanding of the holographic communication methods provided in the above embodiments, the following will be combined with... Figures 7 to 9 The interaction process between the acquisition end, the server end, and the receiving end will be described using a specific embodiment.

[0116] like Figure 7 As shown, the acquisition end performs audio acquisition and multi-view image acquisition on the first object. The multi-view image acquisition includes: using multiple cameras to synchronously acquire images of the first object from different perspectives based on timestamp synchronization technology, obtaining multiple first images, and stitching these multiple first images together to obtain a panoramic image; furthermore, using a first SDK (Software Development Kit), the acquired audio signal and panoramic image are encoded to obtain a second data stream, which is then sent to the server.

[0117] The server receives a second data stream based on RTMP (Real-Time Messaging Protocol). By decoding, preprocessing, and splitting the second data stream, multiple first images (such as image 1, image 2, etc.) are obtained. For each first image, foreground and background separation is performed, and the background of the first image is masked to obtain a corresponding mask image. Further, depth estimation is performed on these mask images to obtain multiple depth images, and attribute prediction is performed on the multiple depth images and the multiple first images to obtain multiple attribute images. Specifically, as shown... Figure 8As shown, the server inputs N mask images into the third model for depth prediction and attribute prediction, resulting in N depth images, N attribute image 1, N attribute image 2, etc. Among them, the attribute data contained in attribute image 1 and attribute image 2 are different. For example, attribute image 1 includes the opacity of multiple primitives, while attribute image 2 includes the scale of multiple primitives.

[0118] Furthermore, the server encodes multiple depth images and multiple attribute images to obtain a first data stream, and uploads the first data stream to the CDN. Through the CDN's node distribution capability, the first data stream is distributed to the receiving end according to the receiving end's access request, so as to ensure the efficiency and stability of transmission.

[0119] The receiving end acquires the first data stream through the second SDK, and obtains a 3D model of the first object based on the first data stream; further, it renders and displays the 3D model in conjunction with the detected viewpoint. Specifically, as follows... Figure 8 As shown, the receiving end performs 3D modeling of the first object based on all depth images and all attribute images, obtains the 3D model of the first object, and renders and displays the 3D model.

[0120] Optionally, the first data stream may also include an audio signal, and the receiving end may also play the audio signal.

[0121] Optionally, the server also designed corresponding lightweight compression strategies for multiple depth images and multiple attribute images. Specifically, such as... Figure 9 As shown, the server quantizes all attribute images based on a first precision and writes the attribute data from each quantized attribute image into the first channel of the corresponding third image to obtain the fifth image. The server also quantizes all depth images based on a second precision and writes the depth values ​​from each quantized depth image into the first and second channels of the corresponding fourth image to obtain the sixth image. The fifth and sixth images are then stitched together to obtain the second image. The second image is then encoded to obtain the first data stream. Finally, the first data stream is sent to the CDN, which distributes it to the receiving end.

[0122] Thus, the deep aggregation of multi-view information ensures the high definition and view integrity of the holographic display; the lightweight compression scheme based on attribute images effectively reduces the transmission bandwidth usage and improves data transmission efficiency; the application of real-time rendering technology on the edge significantly reduces the display latency of the holographic image, ultimately achieving high-quality holographic communication and interaction under low latency and low bandwidth conditions.

[0123] It is worth noting that in practical applications, the above-mentioned interaction process can be implemented by the client and server at the acquisition end, and the client at the receiving end. For example, at the acquisition end, the client triggers audio acquisition, multi-view image acquisition, and encodes the acquired audio signals and multi-view images before sending them to the server. At the receiving end, the client can obtain a 3D model of the first object based on the first data stream, and then render and display the 3D model.

[0124] Figure 10 This is a schematic diagram of the structure of an electronic device provided in one embodiment. Please refer to it. Figure 10 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0125] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0126] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0127] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a holographic communication device at the logical level. The processor executes the program stored in memory and specifically performs the following operations: A three-dimensional model of a first object is obtained; the three-dimensional model is obtained based on multiple depth images and multiple attribute images, the multiple depth images and multiple attribute images are obtained based on multiple first images, the multiple first images are acquired from multiple perspectives of the first object, the multiple attribute images include attribute data of multiple primitives under the multiple perspectives, and the primitives are basic units representing the first object; The 3D model is rendered and displayed based on the client's perspective.

[0128] Alternatively, the processor executes the program stored in memory and specifically performs the following operations: Receive a second data stream, the second data stream including a plurality of first images, the plurality of first images being acquired from multiple perspectives of a first object; Depth estimation is performed on the plurality of first images to obtain a plurality of depth images; Based on the plurality of first images and the plurality of depth images, a plurality of attribute images are obtained. The plurality of attribute images include attribute data of a plurality of primitives under the plurality of viewpoints. The primitives are basic units representing the first object. Based on the plurality of depth images and the plurality of attribute images, a first data stream is obtained and sent to the client. The first data stream is used to render and display the three-dimensional model of the first object.

[0129] The above is as described in this instruction manual. Figure 2 or Figure 4The method executed by the holographic communication device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this specification can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0130] It should be understood that the electronic devices in the embodiments of this specification can realize holographic communication devices. Figure 2 or Figure 4 The embodiments shown have the same function. Since the principle is the same, the embodiments in this specification will not be described again here.

[0131] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] This specification also proposes a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a holographic communication method, such as that applied to a client or a holographic communication method, such as that applied to a server.

[0133] This specification also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps in a holographic communication method applied to a client, or operable to cause a computer to perform some or all of the steps in a holographic communication method applied to a server.

[0134] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0135] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0136] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0137] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A holographic communication method, applied to a client, comprising: Obtain the 3D model of the first object; The three-dimensional model is obtained based on multiple depth images and multiple attribute images. The multiple depth images and multiple attribute images are obtained based on multiple first images. The multiple first images are obtained by acquiring the first object from multiple perspectives. The multiple attribute images include attribute data of multiple primitives under the multiple perspectives. The primitives are the basic units representing the first object. The 3D model is rendered and displayed based on the client's perspective.

2. The method according to claim 1, wherein obtaining the three-dimensional model of the first object comprises: Receive a first data stream, the first data stream including the plurality of depth images and the plurality of attribute images, and back-project the plurality of depth images and the plurality of attribute images into a three-dimensional space to obtain the three-dimensional model; or, The system receives a first data stream and decompresses it to obtain the 3D model. The first data stream is obtained by compressing the 3D model after the server reconstructs it based on the multiple depth images and the multiple attribute images; or... The system receives a first data stream, which includes the plurality of first images, and performs depth estimation on the plurality of first images to obtain the plurality of depth images. Based on the plurality of first images and the plurality of depth images, the system obtains the plurality of attribute images and back-projects the plurality of depth images and the plurality of attribute images into a three-dimensional space to obtain the three-dimensional model.

3. The method according to claim 1, wherein the three-dimensional model comprises the plurality of primitives, and the attribute data of the primitives includes the color of the primitives; The rendering and display of the 3D model based on the 3D model and the client's perspective includes: Based on the client's perspective, the multiple primitives are projected onto the client's two-dimensional imaging plane to obtain the number of pixels covered by each primitive on the two-dimensional imaging plane and the depth value of each primitive. For each pixel on the two-dimensional imaging plane, the color of the at least one first primitive is mixed based on the depth value of the at least one first primitive covering the pixel to obtain a rendered image from the client's perspective. The rendered image is displayed.

4. The method according to claim 3, wherein for each pixel on the two-dimensional imaging plane, mixing the color of the at least one first primitive based on a depth value of at least one first primitive covering the pixel comprises: The two-dimensional imaging plane is divided into multiple regions; For each region, the elements covering the region are sorted based on their depth values ​​to obtain the arrangement order of the multiple elements within the region; For each pixel within the region, the color of the at least one first graphic element is superimposed onto the pixel based on the arrangement order of the at least one first graphic element within the region.

5. The method according to claim 1, wherein the primitives include Gaussian primitives, and the attribute data includes at least one of the following: opacity, scale, shape, position, and color.

6. A holographic communication method, applied to a server, comprising: Receive a second data stream, the second data stream including a plurality of first images, the plurality of first images being acquired from multiple perspectives of a first object; Depth estimation is performed on the plurality of first images to obtain a plurality of depth images; Based on the plurality of first images and the plurality of depth images, a plurality of attribute images are obtained. The plurality of attribute images include attribute data of a plurality of primitives under the plurality of viewpoints. The primitives are basic units representing the first object. Based on the plurality of depth images and the plurality of attribute images, a first data stream is obtained and sent to the client. The first data stream is used to render and display the three-dimensional model of the first object.

7. The method according to claim 6, wherein the first data stream comprises the plurality of depth images and the plurality of attribute images; or, The first data stream is obtained by back-projecting the multiple depth images and the multiple attribute images into a three-dimensional space to obtain the three-dimensional model, and then compressing the three-dimensional model.

8. The method according to claim 6, wherein performing depth estimation on the plurality of first images to obtain a plurality of depth images includes: Feature extraction is performed on the plurality of first images to obtain a plurality of feature images, which correspond to the plurality of first images; The multiple feature images are matched to obtain the pixel correspondence of the multiple first images; Based on the pixel correspondence, the disparity of each pixel in the first image is obtained; The disparity of the pixel is converted into the depth value of the pixel to obtain a depth image corresponding to each first image.

9. The method according to claim 6, further comprising, before performing depth estimation on the plurality of first images to obtain a plurality of depth images: For each first image, perform foreground and background separation on the first image; The background of the first image is masked.

10. The method according to claim 6, wherein the first data stream is generated based on the plurality of depth images and the plurality of attribute images after masking; the plurality of attribute images and the plurality of depth images correspond to the plurality of first images; The masked attribute images are obtained as follows: For each first image, the first image and the corresponding depth image are input into the first model to obtain the redundant region of the first image; The attribute data of the first primitive in the attribute image corresponding to the first image is masked to obtain the masked attribute image, wherein the first primitive corresponds to the pixel in the redundant region.

11. The method according to claim 6, wherein the first data stream is obtained in the following manner: The plurality of attribute images are quantized based on a first precision, wherein the first precision is less than the precision of the plurality of attribute images; The plurality of depth images are quantized based on a second precision, wherein the second precision is less than the precision of the plurality of depth images; The quantized multiple attribute images and the quantized multiple depth images are stitched together to obtain a second image; The second image is encoded to obtain the first data stream.

12. The method according to claim 11, wherein stitching together the quantized plurality of attribute images and the quantized plurality of depth images to obtain a second image comprises: Generate multiple third images and multiple fourth images, wherein the multiple third images correspond to the multiple quantized attribute images, and the multiple fourth images correspond to the multiple quantized depth images; For each quantized attribute image, the attribute data of each primitive in the attribute image is written into the first channel of the third image corresponding to the attribute image to obtain the fifth image; For each quantized depth image, the depth value of each pixel in the depth image is written into the second and third channels of the corresponding fourth image to obtain the sixth image; The fifth image and the sixth image are stitched together to obtain the second image.

13. An electronic device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the holographic communication method as described in any one of claims 1 to 5 or the holographic communication method as described in any one of claims 6 to 11.

14. A computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the holographic communication method as described in any one of claims 1 to 5; or, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the holographic communication method as described in any one of claims 6 to 11.

15. A computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the holographic communication method as claimed in any one of claims 1 to 5; or, the computer program operable to cause a computer to perform the holographic communication method as claimed in any one of claims 6 to 11.

Images