Point cloud compression model training method, point cloud compression method and device

Abstract

The invention relates to the technical field of visual data processing, and provides a training method of a point cloud compression model, and a point cloud compression method and device. The method comprises the following steps: acquiring point cloud data; inputting the point cloud data into a layered encoder in the initial point cloud compression model to obtain a compressed encoding code stream output by the layered encoder; the hierarchical encoder is used for performing video encoding and geometric structure encoding on the point cloud data in parallel to obtain a compressed encoding code stream after code stream merging; inputting the compressed encoding code stream into a decoder in the initial point cloud compression model to obtain point cloud training data output by the decoder; and training the initial point cloud reconstruction model in the decoder based on the point cloud training data to obtain a trained point cloud reconstruction model and a point cloud compression model. According to the embodiment of the invention, joint modeling and compression of the geometric structure and the attribute of the point cloud can be realized, the overall compression efficiency and reconstruction consistency are improved, and the compression performance of the point cloud is improved.

Description

Technical Field

[0001] This application relates to the field of visual data processing technology, and in particular to a training method for a point cloud compression model, a point cloud compression method, and an apparatus. Background Technology

[0002] Point cloud data, as an important form of 3D visual data, is widely used in technologies such as Virtual Reality (VR), Augmented Reality (AR), autonomous driving, and immersive communication. Point cloud data consists of multiple unordered and irregularly distributed 3D coordinate points, each of which can be associated with attribute information such as color and normals. Due to the massive volume of point cloud data, efficient point cloud compression (PCC) technology has become a research focus.

[0003] In existing technologies, point cloud compression is generally performed using a serial encoding paradigm that prioritizes geometry over attributes, or geometry or attributes are encoded independently. This separation of geometry and attributes results in low compression efficiency and affects compression performance. Summary of the Invention

[0004] This application provides a training method, a point cloud compression method, and an apparatus for a point cloud compression model, which solves the defects in the prior art where geometry and attribute encoding are separated during point cloud compression, resulting in low compression efficiency and affecting compression performance.

[0005] In a first aspect, embodiments of this application provide a training method for a point cloud compression model, comprising the following steps.

[0006] Acquire point cloud data; The point cloud data is input into the layered encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the layered encoder; the layered encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain the compressed encoded bitstream after merging the bitstream. The compressed encoded bitstream is input into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder; the initial point cloud reconstruction model in the decoder is trained based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

[0007] In one embodiment, the layered encoder includes a two-dimensional texture encoding module and a three-dimensional structure encoding module; the compressed encoded bitstream includes a two-dimensional video encoded stream and a three-dimensional structure encoded stream. The step of inputting the point cloud data into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder includes: The point cloud data is input into the two-dimensional texture encoding module to obtain a two-dimensional video encoded stream output by the two-dimensional texture encoding module; the two-dimensional texture encoding module is used to perform video encoding on the texture image in the point cloud data based on preset camera parameters; The point cloud data is input into the three-dimensional structure encoding module to obtain the three-dimensional structure encoding stream output by the three-dimensional structure encoding module; the three-dimensional structure encoding module is used to encode the sparse point cloud in the point cloud data; the sparse point cloud is used to characterize the key geometric structural features in the point cloud data.

[0008] In one embodiment, the two-dimensional texture encoding module includes an encoding normalization module, a multi-view projection module, and a video compression module; The step of inputting the point cloud data into the two-dimensional texture encoding module to obtain the two-dimensional video encoded stream output by the two-dimensional texture encoding module includes: The point cloud data is input into the encoding normalization module to obtain the normalized point cloud data output by the encoding normalization module; The normalized point cloud data is input into the multi-view projection module to obtain the texture image output by the multi-view projection module; the multi-view projection module is used to project the normalized point cloud data onto a two-dimensional plane from multiple camera perspectives based on the preset camera parameters. The texture image and the preset camera parameters are input into the video compression module to obtain a two-dimensional video encoded stream output by the video compression module; the video compression module is used to encode the texture image to obtain a video stream, and to encode and compress the preset camera parameters to obtain a parameter stream; the video stream and the parameter stream are determined as the two-dimensional video encoded stream.

[0009] In one embodiment, the three-dimensional structure encoding module includes a keyframe selection module, a downsampling module, and a structure encoder; The step of inputting the point cloud data into the 3D structure encoding module to obtain the 3D structure encoding stream output by the 3D structure encoding module includes: The point cloud data is input into the key frame selection module to obtain key frame point cloud data and non-key frame point cloud data output by the key frame selection module; the key frame selection module is used to divide the point cloud data into key frame point cloud data and non-key frame point cloud data based on relative transformation. The keyframe point cloud data and the non-keyframe point cloud data are input into the downsampling module to obtain a sparse point cloud output by the downsampling module; the downsampling ratios corresponding to the keyframe point cloud data and the non-keyframe point cloud data are different. The sparse point cloud is input into the structure encoder to obtain a three-dimensional structure encoding stream output by the structure encoder; the structure encoder is used to encode the sparse point cloud.

[0010] In one embodiment, the decoder includes a video decoding module, a point cloud decoding module, and an initial point cloud reconstruction model; The step of inputting the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder includes: The two-dimensional video encoded stream is input into the video decoding module to obtain a multi-view texture image and preset camera parameters output by the video decoding module; The three-dimensional structure encoding stream is input into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module; The multi-view texture image, preset camera parameters, and sparse point cloud location information are determined as the point cloud training data input to the initial point cloud reconstruction model.

[0011] In one embodiment, the video decoding module includes a video decoder and a video super-resolution module; The step of inputting the two-dimensional video encoded stream into the video decoding module to obtain the multi-view texture image and preset camera parameters output by the video decoding module includes: The two-dimensional video encoded stream is input into the video decoder to obtain the decoded texture image and preset camera parameters output by the video decoder; the video decoder is used to decode the two-dimensional video encoded stream. The decoded texture image is input into the video super-resolution module to obtain a multi-view texture image output by the video super-resolution module; the video super-resolution module is used to restore texture details of the decoded texture image based on the video super-resolution mechanism to obtain a repaired multi-view texture image.

[0012] In one embodiment, the point cloud decoding module includes a point cloud decoder, a decoding normalization module, and a structure initialization module; The step of inputting the three-dimensional structure encoding stream into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module includes: The three-dimensional structure encoding stream is input into the point cloud decoder to obtain the decoded sparse point cloud output by the point cloud decoder; The decoded sparse point cloud is input into the decoding normalization module to obtain the normalized sparse point cloud output by the decoding normalization module; the decoding normalization module is used to perform coordinate scaling on the decoded sparse point cloud. The normalized sparse point cloud is input into the structure initialization module to obtain the sparse point cloud location information output by the structure initialization module.

[0013] Secondly, embodiments of this application provide a point cloud compression method, including the following steps.

[0014] Obtain target view parameters; The target viewpoint parameters are input into the point cloud compression model to obtain the target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described above. Based on the reconstructed image from the target viewpoint and the preset camera parameters, the target depth map corresponding to the reconstructed image from the target viewpoint is determined; Based on the target depth map, a 3D consistency model is performed to obtain the target point cloud reconstruction data corresponding to the target viewpoint parameters.

[0015] Thirdly, embodiments of this application provide a training apparatus for a point cloud compression model, comprising the following units.

[0016] The first acquisition unit is used to acquire point cloud data.

[0017] The encoding unit is used to input the point cloud data into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder; the hierarchical encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain the compressed encoded bitstream after merging the bitstream.

[0018] The decoding unit is used to input the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder; and to train the initial point cloud reconstruction model in the decoder based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

[0019] Fourthly, embodiments of this application provide a point cloud compression device, including the following units.

[0020] The second acquisition unit is used to acquire the target view parameters.

[0021] The reconstruction unit is used to input the target viewpoint parameters into the point cloud compression model to obtain the target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described above.

[0022] The determining unit is used to determine the target depth map corresponding to the target view reconstructed image based on the target view reconstructed image and preset camera parameters.

[0023] The modeling unit is used to perform three-dimensional consistent modeling based on the target depth map to obtain target point cloud reconstruction data corresponding to the target viewpoint parameters.

[0024] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method for the point cloud compression model as described in the first aspect above, or the point cloud compression method as described in the second aspect.

[0025] In a sixth aspect, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the point cloud compression model as described in the first aspect above, or the point cloud compression method as described in the second aspect.

[0026] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the training method for the point cloud compression model as described in the first aspect above, or the point cloud compression method as described in the second aspect.

[0027] The point cloud compression model training method, point cloud compression method, and apparatus provided in this application embodiment involve inputting acquired point cloud data into a layered encoder in an initial point cloud compression model. The layered encoder performs video encoding and geometric structure encoding on the point cloud data to obtain a compressed encoded bitstream after merging. This compressed encoded bitstream is then input into a decoder in the initial point cloud compression model to obtain decoded point cloud training data. Based on the point cloud training data, an initial point cloud reconstruction model constructed in the decoder using a 3D Gaussian splashing algorithm is trained until the model converges, resulting in a trained point cloud reconstruction model and a point cloud compression model. In this application embodiment, point cloud data undergoes video encoding and geometric structure encoding separately, and is compressed in parallel. This approach is compatible with existing video and point cloud encoding tools and facilitates subsequent perception optimization. The video encoding portion emphasizes visual quality, differing from the traditional pure data fidelity-oriented approach, and is more in line with the user's visual experience. Meanwhile, the introduction of 3D Gaussian splashing into point cloud reconstruction enables joint modeling and compression of point cloud geometry and attributes, more effectively capturing the correlation between geometry and attributes. This breaks the traditional serial coding paradigm of geometry first and attributes later, avoids redundant expressions, and improves overall compression efficiency and reconstruction consistency through joint optimization, thereby enhancing compression performance. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the training method for the point cloud compression model provided in this application embodiment.

[0030] Figure 2 This is one of the structural schematic diagrams of the layered encoder provided in the embodiments of this application.

[0031] Figure 3 This is the second schematic diagram of the structure of the layered encoder provided in the embodiments of this application.

[0032] Figure 4 This is one of the structural schematic diagrams of the decoder provided in the embodiments of this application.

[0033] Figure 5 This is the second schematic diagram of the decoder provided in the embodiments of this application.

[0034] Figure 6 This is a flowchart illustrating the point cloud compression method provided in the embodiments of this application.

[0035] Figure 7 This is a schematic diagram of the structure of the training device for the point cloud compression model provided in the embodiments of this application.

[0036] Figure 8 This is a schematic diagram of the point cloud compression device provided in the embodiments of this application.

[0037] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] To address the problem in existing point cloud compression technologies where geometry and attributes are encoded separately, resulting in low compression efficiency and negatively impacting the final reconstruction quality, this application provides a training method for a point cloud compression model. Figure 1 This is a flowchart illustrating the training method for the point cloud compression model provided in this application embodiment, as shown below. Figure 1 As shown, the method includes steps 110 to 130.

[0040] Step 110: Obtain point cloud data.

[0041] Point cloud data is a set of three-dimensional coordinate points representing a target object or target scene in a three-dimensional coordinate system. The three-dimensional coordinate points in this set are unordered and irregularly distributed, and each three-dimensional coordinate point includes, but is not limited to, attributes such as three-dimensional coordinates, color (Red, Green, Blue, RGB), transparency, and normals.

[0042] In this embodiment of the application, the point cloud data can be obtained from dynamic or static scene data collected by devices such as 3D scanners, LiDAR, or depth cameras. That is, the point cloud data can be dynamic point cloud or static point cloud.

[0043] Step 120: Input the point cloud data into the layered encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the layered encoder; the layered encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain the compressed encoded bitstream after merging the bitstream.

[0044] Specifically, after acquiring point cloud data, the data is input into a layered encoder with a two-layer structure in the initial point cloud compression model. On one hand, the layered encoder performs video encoding on the texture image corresponding to the point cloud data. On the other hand, it extracts sparse point clouds from the point cloud data and encodes them using geometric coding techniques. These two streams are then merged to form the final compressed encoded stream. The compressed encoded stream has a much smaller data volume than the original point cloud data, facilitating storage and network transmission.

[0045] Step 130: Input the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder. Train the initial point cloud reconstruction model in the decoder based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

[0046] Specifically, after determining the compressed encoded bitstream, the compressed encoded bitstream is input into the decoder. The decoder separates the compressed encoded bitstream by inversely processing the hierarchical structure of the hierarchical encoder, recovers the texture image, sparse point cloud, and preset camera parameters after repairing the compression loss from the compressed encoded bitstream, and uses the texture image, sparse point cloud, and preset camera parameters after repairing the compression loss as point cloud training data.

[0047] After determining the point cloud training data, the initial point cloud reconstruction model in the decoder is trained using this data. This initial point cloud reconstruction model is a Gaussian model constructed based on the 3D Gaussian splashing method. By adjusting the Gaussian parameters in the initial point cloud reconstruction model using the point cloud training data, the model minimizes the difference between the rendered 2D image and the decoded texture image from the corresponding viewpoint when rendering the image from a specific perspective, thus achieving point cloud reconstruction based on 3D Gaussian splashing. Based on this, the trained point cloud compression model can be obtained.

[0048] The point cloud compression model training method provided in this application involves inputting the acquired point cloud data into a layered encoder in an initial point cloud compression model. The layered encoder performs video encoding and geometric structure encoding on the point cloud data to obtain a compressed encoded bitstream after merging. This compressed encoded bitstream is then input into a decoder in the initial point cloud compression model to obtain decoded point cloud training data. Based on the point cloud training data, an initial point cloud reconstruction model constructed using a 3D Gaussian splashing algorithm is trained in the decoder until the model converges, resulting in a trained point cloud reconstruction model and a point cloud compression model. In this application embodiment, the point cloud data is subjected to video encoding and geometric structure encoding separately, and then jointly compressed in parallel. This approach is compatible with existing video and point cloud encoding tools and facilitates subsequent perceptual optimization. The video encoding portion emphasizes visual quality, which differs from the traditional pure data fidelity-oriented approach and is more in line with the user's visual experience. Meanwhile, the introduction of 3D Gaussian splashing into point cloud reconstruction enables joint modeling and compression of point cloud geometry and attributes, more effectively capturing the correlation between geometry and attributes. This breaks the traditional serial coding paradigm of geometry first and attributes later, avoids redundant expressions, and improves overall compression efficiency and reconstruction consistency through joint optimization, thereby enhancing compression performance.

[0049] It should be noted that existing technologies often aim for high bitrates, which significantly degrades visual quality at low bitrates, especially in multi-user scenarios on mobile networks where bandwidth often falls short. The embodiments of this application, however, are suitable for low bitrate transmission scenarios. Through layered coding and point cloud reconstruction based on 3D Gaussian splashing, high-fidelity compression driven by visual perception is achieved at low bitrates. The embodiments of this application, even under resource constraints, better preserve key geometric structures and attribute features, achieving higher visual quality. These embodiments effectively meet the practical needs of lightweight transmission and resource-constrained terminal devices for point cloud compression.

[0050] In one embodiment, Figure 2 This is a schematic diagram of the structure of the layered encoder provided in the embodiments of this application, as shown below. Figure 2 As shown, the layered encoder includes a two-dimensional texture encoding module and a three-dimensional structure encoding module; the compressed encoded bitstream includes a two-dimensional video encoded stream and a three-dimensional structure encoded stream. The step of inputting the point cloud data into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder includes: The point cloud data is input into the two-dimensional texture encoding module to obtain a two-dimensional video encoded stream output by the two-dimensional texture encoding module; the two-dimensional texture encoding module is used to perform video encoding on the texture image in the point cloud data based on preset camera parameters; The point cloud data is input into the three-dimensional structure encoding module to obtain the three-dimensional structure encoding stream output by the three-dimensional structure encoding module; the three-dimensional structure encoding module is used to encode the sparse point cloud in the point cloud data; the sparse point cloud is used to characterize the key geometric structural features in the point cloud data.

[0051] Specifically, after acquiring point cloud data, the data is input into a 2D texture encoding module. This module extracts texture images representing human visual perception from the original point cloud data and encodes and transmits them in compressed form. This significantly improves the visual reproduction quality of the point cloud data at the rendering end, making it particularly suitable for low bitrate transmission scenarios. Within this 2D texture encoding module, the texture image corresponding to the point cloud data is extracted by combining preset camera parameters. Then, the texture image is combined with the preset camera parameters and encoded to obtain a 2D video encoded stream. This stream is compatible with existing video encoding frameworks and enables low bitrate compression.

[0052] On the other hand, the point cloud data is input into a 3D structure encoding module. This module abstracts and simplifies the spatial structure of the point cloud data before compression and compresses and stores it using point cloud encoding, ensuring accurate reconstruction of the overall geometric contour of the point cloud data even in low-bandwidth environments. Within this 3D structure encoding module, the point cloud data is downsampled to extract sparse point clouds. These sparse point clouds are then encoded to obtain a 3D structure encoded stream, enabling integration with existing encoding frameworks and preserving the key geometric structure of the point cloud in low-bit-rate scenarios.

[0053] In one embodiment, the two-dimensional texture encoding module includes an encoding normalization module, a multi-view projection module, and a video compression module; The step of inputting the point cloud data into the two-dimensional texture encoding module to obtain the two-dimensional video encoded stream output by the two-dimensional texture encoding module includes: The point cloud data is input into the encoding normalization module to obtain the normalized point cloud data output by the encoding normalization module; The normalized point cloud data is input into the multi-view projection module to obtain the texture image output by the multi-view projection module; the multi-view projection module is used to project the normalized point cloud data onto a two-dimensional plane from multiple camera perspectives based on the preset camera parameters. The texture image and the preset camera parameters are input into the video compression module to obtain a two-dimensional video encoded stream output by the video compression module; the video compression module is used to encode the texture image to obtain a video stream, and to encode and compress the preset camera parameters to obtain a parameter stream; the video stream and the parameter stream are determined as the two-dimensional video encoded stream.

[0054] Specifically, Figure 3 This is a second schematic diagram of the structure of the layered encoder provided in the embodiments of this application, as shown below. Figure 3 As shown, after inputting the point cloud data into the two-dimensional texture encoding module, the two-dimensional texture encoding module is then input into the encoding normalization module. The encoding normalization module performs centering and scale unification operations on the point cloud data, transforming the coordinates of all three-dimensional coordinate points in the point cloud data and normalizing the coordinates to a preset range, preferably [-1, 1], to eliminate scale differences between different scenes and improve geometric stability and image consistency in the subsequent texture image determination process. After normalizing the point cloud data, normalized point cloud data is obtained.

[0055] Next, the normalized point cloud data is input into a multi-view projection module. This module is configured with a set of preset camera parameters, including virtual camera intrinsic and extrinsic parameters. For example, the virtual camera intrinsic parameters include focal length and field of view, while the virtual camera extrinsic parameters include rotation matrix and translation vector. In this multi-view projection module, using these preset camera parameters and models such as weak perspective, the normalized point cloud data is projected onto a two-dimensional plane from multiple virtual camera perspectives. This simulates capturing the normalized point cloud data from multiple different viewpoints, resulting in multiple texture images containing rich color information. These texture images reflect the texture characteristics of the normalized point cloud data.

[0056] After obtaining the texture image, the texture image and preset camera parameters are input into the video compression module. In this module, the generated multi-view texture image sequence is uniformly encoded into a video stream by a video encoder. Simultaneously, the preset camera parameters involved in the projection process are encoded and compressed to obtain a parameter stream; for example, the rotation matrix is ​​converted into quaternions to reduce redundancy. Then, the parameter stream is compressed as Supplemental Enhancement Information (SEI) of the video encoder to reduce transmission load, thereby obtaining a two-dimensional video encoded stream.

[0057] It should be noted that when encoding texture images for video, various mainstream video codecs (i.e., video encoders) such as HEVC (High Efficiency Video Coding) and VCC (Versatile Video Coding) can be used to integrate with the existing compression ecosystem and reduce development and deployment costs.

[0058] It should be noted that the number and distribution of virtual camera viewpoints in the multi-view projection module can be flexibly configured to adapt to the needs of different scenarios, and this application embodiment does not limit this.

[0059] In one embodiment, the three-dimensional structure encoding module includes a keyframe selection module, a downsampling module, and a structure encoder; The step of inputting the point cloud data into the 3D structure encoding module to obtain the 3D structure encoding stream output by the 3D structure encoding module includes: The point cloud data is input into the key frame selection module to obtain key frame point cloud data and non-key frame point cloud data output by the key frame selection module; the key frame selection module is used to divide the point cloud data into key frame point cloud data and non-key frame point cloud data based on relative transformation. The keyframe point cloud data and the non-keyframe point cloud data are input into the downsampling module to obtain a sparse point cloud output by the downsampling module; the downsampling ratios corresponding to the keyframe point cloud data and the non-keyframe point cloud data are different. The sparse point cloud is input into the structure encoder to obtain a three-dimensional structure encoding stream output by the structure encoder; the structure encoder is used to encode the sparse point cloud.

[0060] Specifically, keyframe point cloud data refers to frames in a point cloud sequence that exhibit significant geometric changes or motion amplitude, preserving the main structural information of the target scene. Relative transformations are used to characterize the relative motion or geometric changes between point clouds. These relative transformations include rotation matrices or translation vectors.

[0061] like Figure 3 As shown, after inputting the point cloud data into the 3D structure encoding module, the point cloud data is then input into the keyframe selection module. In this keyframe selection module, the first frame of the point cloud data can be used as the first keyframe. Through point cloud registration, the transformation amount between the current frame (i.e., the second frame) and the previous keyframe (i.e., the first frame) is calculated, and the degree of geometric change is evaluated using this transformation amount. If the transformation amount is greater than a preset threshold, it indicates a large geometric change between the current frame and the previous keyframe, and the current frame can be identified as keyframe point cloud data. If the transformation amount is less than or equal to a preset threshold, it indicates a small geometric change between the current frame and the previous keyframe, and the current frame can be identified as non-keyframe point cloud data. Then, the next frame is identified as the current frame, and the above operation is repeated to classify the new current frame, i.e., to determine whether the new current frame is keyframe point cloud data or non-keyframe point cloud data. This operation is repeated to classify all point cloud data, resulting in keyframe point cloud data and non-keyframe point cloud data.

[0062] It should be noted that point cloud registration refers to the process of aligning the current frame with the previous keyframe to the same coordinate system. The preferred point cloud registration algorithm is the Iterative Closest Point (ICP) algorithm, which minimizes the error between point clouds by iteratively optimizing rotation and translation transformations.

[0063] Next, keyframe point cloud data and non-keyframe point cloud data are input into the downsampling module. The downsampling module uses point cloud sampling methods and different downsampling ratios to downsample both keyframe and non-keyframe point cloud data. Keyframe point cloud data contains important geometric structure information; therefore, a smaller downsampling ratio is used in the downsampling module to retain more representative 3D coordinate points. Non-keyframe point cloud data has less geometric variation, so a larger downsampling ratio is used. Based on this, a sparse point cloud corresponding to the point cloud data is obtained. The 3D coordinate points in this sparse point cloud retain the main spatial morphology of the point cloud data, can adapt to different point density distributions, and improve sampling robustness and compression efficiency.

[0064] Optionally, the point cloud sampling method can be a regular sampling method or a density-based adaptive sampling method, etc. The regular sampling method can be a voxel grid sampling method, and this application embodiment does not limit it.

[0065] Optionally, the downsampling ratios corresponding to keyframe point cloud data and non-keyframe point cloud data can be dynamically adjusted as needed, ensuring that the downsampling ratio corresponding to keyframe point cloud data is less than the downsampling ratio corresponding to non-keyframe point cloud data. This application embodiment does not limit this.

[0066] After determining the sparse point cloud, it is input into a structure encoder. This encoder uses a standardized point cloud encoding tool to sequentially perform operations such as octree partitioning, quantization, and entropy encoding on the sparse point cloud. Octtree partitioning recursively divides the disordered and scattered coordinate space corresponding to the sparse point cloud into regular cubes, generating a tree structure and establishing a spatial hierarchy of the sparse point cloud, achieving efficient spatial organization and hierarchical representation. Quantization maps the high-precision floating-point coordinates of the 3D coordinates in each cube to a finite-precision integer grid, reducing the amount of data required to represent each 3D coordinate point in the sparse point cloud and controlling data precision and volume. Entropy encoding utilizes the statistical characteristics of the data to perform lossless compression of the sparse point cloud based on information theory according to the frequency of occurrence, obtaining a 3D structured encoded stream.

[0067] It should be noted that quantization allows for flexible adjustment of the compression bit rate, achieving a balance between performance and accuracy.

[0068] Optionally, the standardized point cloud encoding tool is preferably G-PCC (Geometry-based Point Cloud Compression), but this application embodiment does not limit it.

[0069] In addition, such as Figure 3 As shown, after the structure encoder obtains the three-dimensional structure encoded stream, it inputs the three-dimensional structure encoded stream into the two-dimensional texture encoding module for merging the bitstreams to obtain the final compressed encoded bitstream.

[0070] In one embodiment, Figure 4 This is one of the structural schematic diagrams of the decoder provided in the embodiments of this application, such as... Figure 4 As shown, the decoder includes a video decoding module, a point cloud decoding module, and an initial point cloud reconstruction model; The step of inputting the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder includes: The two-dimensional video encoded stream is input into the video decoding module to obtain a multi-view texture image and preset camera parameters output by the video decoding module; The three-dimensional structure encoding stream is input into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module; The multi-view texture image, preset camera parameters, and sparse point cloud location information are determined as the point cloud training data input to the initial point cloud reconstruction model.

[0071] Specifically, after receiving the compressed encoded bitstream, the decoder decodes two streams from it. One decoded 2D video encoded stream is input into the video decoding module to obtain a decoded low-resolution multi-view texture image and preset camera parameters. The other decoded 3D structure encoded stream is input into the point cloud decoding module to obtain decoded sparse point cloud position information, which represents the positions of 3D coordinate points. Then, the two decoded multi-view texture images, preset camera parameters, and sparse point cloud position information are all used as point cloud training data to train the initial point cloud reconstruction model in the decoder.

[0072] In one embodiment, Figure 5 This is a second schematic diagram of the decoder structure provided in the embodiments of this application, as shown below. Figure 5 As shown, the video decoding module includes a video decoder and a video super-resolution module; The step of inputting the two-dimensional video encoded stream into the video decoding module to obtain the multi-view texture image and preset camera parameters output by the video decoding module includes: The two-dimensional video encoded stream is input into the video decoder to obtain the decoded texture image and preset camera parameters output by the video decoder; the video decoder is used to decode the two-dimensional video encoded stream. The decoded texture image is input into the video super-resolution module to obtain a multi-view texture image output by the video super-resolution module; the video super-resolution module is used to restore texture details of the decoded texture image based on the video super-resolution mechanism to obtain a repaired multi-view texture image.

[0073] Specifically, after decoding the 2D video encoded stream from the compressed encoded bitstream, this 2D video encoded stream is input into a video decoder. The video decoder decodes the 2D video encoded stream to obtain a low-resolution decoded texture image and preset camera parameters. During the point cloud encoding and compression process, some texture details are lost; therefore, the decoded texture image differs from the unencoded texture image. After obtaining the decoded texture image, it is input into a video super-resolution module. In this module, a video super-resolution mechanism is used to restore details from the low-resolution decoded texture image, recovering the texture details lost due to compression, improving the resolution and frame quality of the decoded texture image, and obtaining a higher-quality, repaired multi-view texture image. This provides more visually reliable input texture frames for the subsequent training of the initial point cloud reconstruction model.

[0074] In this embodiment of the application, in order to address the problem of insufficient compressed image quality, an image super-resolution mechanism is introduced to improve image details, enhance visual quality, effectively compensate for the lack of texture information caused by low bit rate, and enhance the stability and expressiveness of subsequent initial point cloud reconstruction model training.

[0075] In one embodiment, such as Figure 5 As shown, the point cloud decoding module includes a point cloud decoder, a decoding normalization module, and a structure initialization module; The step of inputting the three-dimensional structure encoding stream into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module includes: The three-dimensional structure encoding stream is input into the point cloud decoder to obtain the decoded sparse point cloud output by the point cloud decoder; The decoded sparse point cloud is input into the decoding normalization module to obtain the normalized sparse point cloud output by the decoding normalization module; the decoding normalization module is used to perform coordinate scaling on the decoded sparse point cloud. The normalized sparse point cloud is input into the structure initialization module to obtain the sparse point cloud location information output by the structure initialization module.

[0076] Specifically, after decoding the 3D structure encoded stream from the compressed encoded bitstream, this 3D structure encoded stream is input into the point cloud decoder to recover the 3D coordinates of the sparse point cloud, resulting in a decoded sparse point cloud. Next, the decoded sparse point cloud is input into a decoding normalization module. This module performs coordinate scaling operations on the decoded sparse point cloud, transforming the coordinates of all 3D coordinate points within the decoded sparse point cloud to a preset range, resulting in a normalized sparse point cloud. The coordinates of the 3D coordinate points in this normalized sparse point cloud maintain a uniform spatial scale with the texture image obtained from the 2D projection; that is, it ensures that the geometric structure in the decoded sparse point cloud is spatially aligned with the decoded multi-view texture image. Then, the normalized sparse point cloud is input into a structure initialization module. In this module, the normalized sparse point cloud is used as the initial geometric structure for the initial point cloud reconstruction model, constructing an initial set of Gaussian point positions to provide initial high-dimensional support for subsequent model training, adoption number optimization, and rendering processes. For example, in the structure initialization module, the position of each three-dimensional coordinate point in the normalized sparse point cloud is parsed, i.e., the sparse point cloud position information, and it is used as the initial center position of the corresponding Gaussian ellipsoid in the initial point cloud reconstruction model.

[0077] After decoding multi-view texture images, preset camera parameters, and sparse point cloud location information, an initial point cloud reconstruction model is initialized based on the sparse point cloud location information. The attributes of each 3D coordinate point are determined as other Gaussian parameters corresponding to each Gaussian ellipsoid; that is, initial parameters such as scale, transparency, and color are set for each Gaussian ellipsoid. Then, the initial point cloud reconstruction model is iteratively trained based on the multi-view texture images and preset camera parameters. The training objective is to minimize the difference between the 2D image projected onto a 2D plane from any camera viewpoint and the corresponding multi-view texture image from that camera viewpoint by iteratively adjusting the initialized Gaussian parameters. This difference can be measured using metrics such as L1 loss, PSNR (Peak signal-to-noise ratio), SSIM (Structural Similarity), and LPILS (Learned Perceptual Image Patch Similarity). Backpropagation is performed based on this difference to update the Gaussian parameters of all Gaussian ellipsoids, enabling them to produce more realistic rendering effects in the next iteration. Furthermore, during training, to improve the efficiency and quality of the initial point cloud reconstruction model, the number of Gaussian ellipsoids is dynamically managed. For example, Gaussian ellipsoids with low contribution are removed, or those with excessively large coverage areas are segmented to increase detail. This process is repeated until the model converges, resulting in a trained point cloud reconstruction model, and subsequently a trained point cloud compression model.

[0078] It should be noted that existing technologies generally emphasize data fidelity rather than perceived quality. In practical applications, such as immersive communication and mixed reality, users primarily interact with rendered 2D images, rather than the original 3D data. Therefore, traditional evaluation methods using 3D geometric fidelity metrics such as coordinate errors can lead to a mismatch between compression results and the actual visual experience. Therefore, this application's embodiments introduce multi-view rendered 2D images for visual quality evaluation, breaking through the traditional point cloud evaluation paradigm that focuses primarily on geometric errors, and better aligning with the perceptual experience in real-world applications. Image domain metrics such as PSNR, SSIM, and LPILS can comprehensively measure the visual fidelity of compressed point clouds, improving the consistency between evaluation results and user subjective experience, thereby providing a more scientific and reasonable evaluation standard for rendering-oriented point cloud coding.

[0079] This application also provides a point cloud compression method. Figure 6 This is a flowchart illustrating the point cloud compression method provided in an embodiment of this application, as shown below. Figure 6 As shown, the method includes steps 610 to 640.

[0080] Step 610: Obtain target view parameters.

[0081] Specifically, in response to user input, the system obtains the target view parameters input by the user. These target view parameters may include parameters such as the target azimuth or the target position.

[0082] Step 620: Input the target viewpoint parameters into the point cloud compression model to obtain the target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described in any of the above embodiments.

[0083] Specifically, after obtaining the target viewpoint parameters, the target viewpoint parameters are input into the trained point cloud compression model. The point cloud compression model uses the differential rendering technology corresponding to the three-dimensional Gaussian splash to project the Gaussian ellipsoid onto the two-dimensional plane from the camera viewpoint corresponding to the target viewpoint parameters, and obtain the rendered target viewpoint reconstruction image.

[0084] Step 630: Based on the target viewpoint reconstructed image and preset camera parameters, determine the target depth map corresponding to the target viewpoint reconstructed image.

[0085] Specifically, after determining the target viewpoint reconstructed image, the depth value of each pixel can be extracted from the target viewpoint reconstructed image by combining preset camera parameters. For example, by combining the three-dimensional coordinates of the Gaussian ellipsoid corresponding to each pixel in the target viewpoint reconstructed image with the preset camera parameters, the depth value corresponding to each pixel can be calculated, thereby obtaining the target depth map corresponding to the target viewpoint reconstructed image.

[0086] Step 640: Perform three-dimensional consistency modeling based on the target depth map to obtain the target point cloud reconstruction data corresponding to the target viewpoint parameters.

[0087] Specifically, after determining the target depth map and the target viewpoint reconstructed image, a 3D consistency model can be performed using the truncated signed distance function (TSDF) or other fusion mechanisms to obtain primary point cloud reconstruction data. Then, this primary point cloud reconstruction data is restored from the normalized space to the original scale coordinate system to obtain the target point cloud reconstruction data corresponding to the target viewpoint parameters, ensuring that the final generated target point cloud reconstruction data maintains the same scene size as the original point cloud data.

[0088] In this embodiment, the powerful application capabilities of the trained point cloud compression model are utilized to achieve real-time and high-quality rendering from any perspective. By fusing multiple views of the target depth map and the target perspective reconstructed image, complete and high-precision 3D point cloud data is reconstructed. The output results are highly flexible and of high quality, which can meet the needs of different applications such as AR (Augmented Reality), VR (Virtual Reality), digital archiving, or reverse engineering.

[0089] The training apparatus for the point cloud compression model provided in the embodiments of this application will be described below. The training apparatus for the point cloud compression model described below and the training method for the point cloud compression model described above can be referred to in correspondence.

[0090] This application provides a training device for a point cloud compression model. Figure 7 This is a schematic diagram of the structure of the training device for the point cloud compression model provided in the embodiments of this application, as shown below. Figure 7 As shown, the training device 700 for the point cloud compression model includes: a first acquisition unit 710, an encoding unit 720, and a decoding unit 730.

[0091] The first acquisition unit 710 is used to acquire point cloud data.

[0092] The encoding unit 720 is used to input the point cloud data into the layered encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the layered encoder; the layered encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain the compressed encoded bitstream after merging the bitstream.

[0093] The decoding unit 730 is used to input the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder; and to train the initial point cloud compression model in the decoder based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

[0094] The point cloud compression model training device provided in this application embodiment inputs the acquired point cloud data into the hierarchical encoder in the initial point cloud compression model. The hierarchical encoder performs video encoding and geometric structure encoding on the point cloud data to obtain a compressed encoded bitstream after merging. The compressed encoded bitstream is then input into the initial point cloud compression model in the decoder to obtain decoded point cloud training data. Based on the point cloud training data, the initial point cloud reconstruction model constructed in the decoder based on the three-dimensional Gaussian splashing algorithm is trained until the model converges, resulting in a trained point cloud reconstruction model and a point cloud compression model. In this application embodiment, the point cloud data is subjected to video encoding and geometric structure encoding separately, and then jointly compressed in parallel. This is compatible with existing video and point cloud encoding tools and facilitates subsequent perception optimization. The video encoding part emphasizes visual quality, which is different from the traditional pure data fidelity orientation and is more in line with the user's visual experience. Meanwhile, the introduction of 3D Gaussian splashing into point cloud reconstruction enables joint modeling and compression of point cloud geometry and attributes, more effectively capturing the correlation between geometry and attributes. This breaks the traditional serial coding paradigm of geometry first and attributes later, avoids redundant expressions, and improves overall compression efficiency and reconstruction consistency through joint optimization, thereby enhancing compression performance.

[0095] Optionally, the layered encoder includes a two-dimensional texture encoding module and a three-dimensional structure encoding module; the compressed encoded bitstream includes a two-dimensional video encoded stream and a three-dimensional structure encoded stream.

[0096] Optionally, the coding unit 720 is specifically used for: The point cloud data is input into the two-dimensional texture encoding module to obtain a two-dimensional video encoded stream output by the two-dimensional texture encoding module; the two-dimensional texture encoding module is used to perform video encoding on the texture image in the point cloud data based on preset camera parameters; The point cloud data is input into the three-dimensional structure encoding module to obtain the three-dimensional structure encoding stream output by the three-dimensional structure encoding module; the three-dimensional structure encoding module is used to encode the sparse point cloud in the point cloud data; the sparse point cloud is used to characterize the key geometric structural features in the point cloud data.

[0097] Optionally, the two-dimensional texture encoding module includes an encoding normalization module, a multi-view projection module, and a video compression module.

[0098] Optionally, the encoding unit 720 is specifically used for: The point cloud data is input into the encoding normalization module to obtain the normalized point cloud data output by the encoding normalization module; The normalized point cloud data is input into the multi-view projection module to obtain the texture image output by the multi-view projection module; the multi-view projection module is used to project the normalized point cloud data onto a two-dimensional plane from multiple camera perspectives based on the preset camera parameters. The texture image and the preset camera parameters are input into the video compression module to obtain a two-dimensional video encoded stream output by the video compression module; the video compression module is used to encode the texture image to obtain a video stream, and to encode and compress the preset camera parameters to obtain a parameter stream; the video stream and the parameter stream are determined as the two-dimensional video encoded stream.

[0099] Optionally, the three-dimensional structure encoding module includes a keyframe selection module, a downsampling module, and a structure encoder.

[0100] Optionally, the coding unit 720 is specifically used for: The point cloud data is input into the key frame selection module to obtain key frame point cloud data and non-key frame point cloud data output by the key frame selection module; the key frame selection module is used to divide the point cloud data into key frame point cloud data and non-key frame point cloud data based on relative transformation. The keyframe point cloud data and the non-keyframe point cloud data are input into the downsampling module to obtain a sparse point cloud output by the downsampling module; the downsampling ratios corresponding to the keyframe point cloud data and the non-keyframe point cloud data are different. The sparse point cloud is input into the structure encoder to obtain a three-dimensional structure encoding stream output by the structure encoder; the structure encoder is used to encode the sparse point cloud.

[0101] Optionally, the decoder includes a video decoding module, a point cloud decoding module, and an initial point cloud reconstruction model.

[0102] Optionally, the decoding unit 730 is specifically used for: The two-dimensional video encoded stream is input into the video decoding module to obtain a multi-view texture image and preset camera parameters output by the video decoding module; The three-dimensional structure encoding stream is input into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module; The multi-view texture image, preset camera parameters, and sparse point cloud location information are determined as the point cloud training data input to the initial point cloud reconstruction model.

[0103] Optionally, the video decoding module includes a video decoder and a video super-resolution module.

[0104] Optionally, the decoding unit 730 is specifically used for: The two-dimensional video encoded stream is input into the video decoder to obtain the decoded texture image and preset camera parameters output by the video decoder; the video decoder is used to decode the two-dimensional video encoded stream. The decoded texture image is input into the video super-resolution module to obtain a multi-view texture image output by the video super-resolution module; the video super-resolution module is used to restore texture details of the decoded texture image based on the video super-resolution mechanism to obtain a repaired multi-view texture image.

[0105] Optionally, the point cloud decoding module includes a point cloud decoder, a decoding normalization module, and a structure initialization module.

[0106] Optionally, the decoding unit 730 is specifically used for: The three-dimensional structure encoding stream is input into the point cloud decoder to obtain the decoded sparse point cloud output by the point cloud decoder; The decoded sparse point cloud is input into the decoding normalization module to obtain the normalized sparse point cloud output by the decoding normalization module; the decoding normalization module is used to perform coordinate scaling on the decoded sparse point cloud. The normalized sparse point cloud is input into the structure initialization module to obtain the sparse point cloud location information output by the structure initialization module.

[0107] This application provides a point cloud compression device. Figure 8 This is a schematic diagram of the point cloud compression device provided in the embodiments of this application, as shown below. Figure 8 As shown, the point cloud compression device 800 includes: a second acquisition unit 810, a reconstruction unit 820, a determination unit 830, and a modeling unit 840.

[0108] The second acquisition unit 810 is used to acquire target view parameters.

[0109] The reconstruction unit 820 is used to input the target viewpoint parameters into the point cloud compression model to obtain the target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described in any of the above embodiments.

[0110] The determining unit 830 is used to determine the target depth map corresponding to the target view reconstructed image based on the target view reconstructed image and preset camera parameters.

[0111] The modeling unit 840 is used to perform three-dimensional consistent modeling based on the target depth map to obtain target point cloud reconstruction data corresponding to the target viewpoint parameters.

[0112] In this embodiment, the powerful application capabilities of the trained point cloud compression model are utilized to achieve real-time and high-quality rendering from any perspective. By fusing multiple views of the target depth map and the target perspective reconstructed image, complete and high-precision 3D point cloud data is reconstructed. The output results are highly flexible and of high quality, which can meet the needs of different applications such as AR (Augmented Reality), VR (Virtual Reality), digital archiving, or reverse engineering.

[0113] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communications bus 940. The processor 910 can call logic instructions in the memory 930 to execute a training method or a point cloud compression method for a point cloud compression model. The training method for the point cloud compression model includes: acquiring point cloud data; inputting the point cloud data into a layered encoder in an initial point cloud compression model to obtain a compressed encoded bitstream output by the layered encoder; the layered encoder performing parallel video encoding and geometric structure encoding on the point cloud data to obtain a compressed encoded bitstream after merging; inputting the compressed encoded bitstream into a decoder in the initial point cloud compression model to obtain point cloud training data output by the decoder; training the initial point cloud compression model in the decoder based on the point cloud training data to obtain a trained point cloud reconstruction model and a point cloud compression model; the initial point cloud reconstruction model is constructed based on a three-dimensional Gaussian splashing algorithm.

[0114] The point cloud compression method includes: acquiring target viewpoint parameters; inputting the target viewpoint parameters into a point cloud compression model to obtain a target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described in any of the above embodiments; determining a target depth map corresponding to the target viewpoint reconstructed image based on the target viewpoint reconstructed image and preset camera parameters; and performing three-dimensional consistency modeling based on the target depth map to obtain target point cloud reconstructed data corresponding to the target viewpoint parameters.

[0115] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method or point cloud compression method of the point cloud compression model provided by the above methods. The training method of the point cloud compression model includes: acquiring point cloud data; inputting the point cloud data into a layered encoder in an initial point cloud compression model to obtain a compressed encoded bitstream output by the layered encoder; the layered encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain a compressed encoded bitstream after merging the bitstream; inputting the compressed encoded bitstream into a decoder in the initial point cloud compression model to obtain point cloud training data output by the decoder; training the initial point cloud compression model in the decoder based on the point cloud training data to obtain a trained point cloud reconstruction model and a point cloud compression model; the initial point cloud reconstruction model is constructed based on a three-dimensional Gaussian splashing algorithm.

[0117] The point cloud compression method includes: acquiring target viewpoint parameters; inputting the target viewpoint parameters into a point cloud compression model to obtain a target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described in any of the above embodiments; determining a target depth map corresponding to the target viewpoint reconstructed image based on the target viewpoint reconstructed image and preset camera parameters; and performing three-dimensional consistency modeling based on the target depth map to obtain target point cloud reconstructed data corresponding to the target viewpoint parameters.

[0118] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method or a point cloud compression method for the point cloud compression model provided by the above methods. The training method for the point cloud compression model includes: acquiring point cloud data; inputting the point cloud data into a layered encoder in an initial point cloud compression model to obtain a compressed encoded bitstream output by the layered encoder; the layered encoder performing parallel video encoding and geometric structure encoding on the point cloud data to obtain a compressed encoded bitstream after merging; inputting the compressed encoded bitstream into a decoder in the initial point cloud compression model to obtain point cloud training data output by the decoder; training the initial point cloud compression model in the decoder based on the point cloud training data to obtain a trained point cloud reconstruction model and a point cloud compression model; the initial point cloud reconstruction model is constructed based on a three-dimensional Gaussian splashing algorithm.

[0119] The point cloud compression method includes: acquiring target viewpoint parameters; inputting the target viewpoint parameters into a point cloud compression model to obtain a target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model described in any of the above embodiments; determining a target depth map corresponding to the target viewpoint reconstructed image based on the target viewpoint reconstructed image and preset camera parameters; and performing three-dimensional consistency modeling based on the target depth map to obtain target point cloud reconstructed data corresponding to the target viewpoint parameters.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a point cloud compression model, characterized in that, include: Acquire point cloud data; The point cloud data is input into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder. The layered encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain a compressed encoded bitstream after merging the bitstream. The compressed encoded bitstream is input into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder; The initial point cloud reconstruction model in the decoder is trained based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

2. The training method for the point cloud compression model according to claim 1, characterized in that, The layered encoder includes a two-dimensional texture encoding module and a three-dimensional structure encoding module; the compressed encoded bitstream includes a two-dimensional video encoded stream and a three-dimensional structure encoded stream. The step of inputting the point cloud data into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder includes: The point cloud data is input into the two-dimensional texture encoding module to obtain a two-dimensional video encoded stream output by the two-dimensional texture encoding module; the two-dimensional texture encoding module is used to perform video encoding on the texture image in the point cloud data based on preset camera parameters; The point cloud data is input into the three-dimensional structure encoding module to obtain the three-dimensional structure encoding stream output by the three-dimensional structure encoding module; the three-dimensional structure encoding module is used to encode the sparse point cloud in the point cloud data; the sparse point cloud is used to characterize the key geometric structural features in the point cloud data.

3. The training method for the point cloud compression model according to claim 2, characterized in that, The two-dimensional texture encoding module includes an encoding normalization module, a multi-view projection module, and a video compression module. The step of inputting the point cloud data into the two-dimensional texture encoding module to obtain the two-dimensional video encoded stream output by the two-dimensional texture encoding module includes: The point cloud data is input into the encoding normalization module to obtain the normalized point cloud data output by the encoding normalization module; The normalized point cloud data is input into the multi-view projection module to obtain the texture image output by the multi-view projection module; the multi-view projection module is used to project the normalized point cloud data onto a two-dimensional plane from multiple camera perspectives based on the preset camera parameters. The texture image and the preset camera parameters are input into the video compression module to obtain a two-dimensional video encoded stream output by the video compression module; the video compression module is used to encode the texture image to obtain a video stream, and to encode and compress the preset camera parameters to obtain a parameter stream; the video stream and the parameter stream are determined as the two-dimensional video encoded stream.

4. The training method for the point cloud compression model according to claim 2, characterized in that, The three-dimensional structure encoding module includes a keyframe selection module, a downsampling module, and a structure encoder; The step of inputting the point cloud data into the 3D structure encoding module to obtain the 3D structure encoding stream output by the 3D structure encoding module includes: The point cloud data is input into the key frame selection module to obtain key frame point cloud data and non-key frame point cloud data output by the key frame selection module; the key frame selection module is used to divide the point cloud data into key frame point cloud data and non-key frame point cloud data based on relative transformation. The keyframe point cloud data and the non-keyframe point cloud data are input into the downsampling module to obtain a sparse point cloud output by the downsampling module; the downsampling ratios corresponding to the keyframe point cloud data and the non-keyframe point cloud data are different. The sparse point cloud is input into the structure encoder to obtain a three-dimensional structure encoding stream output by the structure encoder; the structure encoder is used to encode the sparse point cloud.

5. The training method for the point cloud compression model according to any one of claims 2-4, characterized in that, The decoder includes a video decoding module, a point cloud decoding module, and an initial point cloud reconstruction model; The step of inputting the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder includes: The two-dimensional video encoded stream is input into the video decoding module to obtain a multi-view texture image and preset camera parameters output by the video decoding module; The three-dimensional structure encoding stream is input into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module; The multi-view texture image, preset camera parameters, and sparse point cloud location information are determined as the point cloud training data input to the initial point cloud reconstruction model.

6. The training method for the point cloud compression model according to claim 5, characterized in that, The video decoding module includes a video decoder and a video super-resolution module; The step of inputting the two-dimensional video encoded stream into the video decoding module to obtain the multi-view texture image and preset camera parameters output by the video decoding module includes: The two-dimensional video encoded stream is input into the video decoder to obtain the decoded texture image and preset camera parameters output by the video decoder; the video decoder is used to decode the two-dimensional video encoded stream. The decoded texture image is input into the video super-resolution module to obtain a multi-view texture image output by the video super-resolution module; the video super-resolution module is used to restore texture details of the decoded texture image based on the video super-resolution mechanism to obtain a repaired multi-view texture image.

7. The training method for the point cloud compression model according to claim 5, characterized in that, The point cloud decoding module includes a point cloud decoder, a decoding normalization module, and a structure initialization module; The step of inputting the three-dimensional structure encoding stream into the point cloud decoding module to obtain the sparse point cloud position information output by the point cloud decoding module includes: The three-dimensional structure encoding stream is input into the point cloud decoder to obtain the decoded sparse point cloud output by the point cloud decoder; The decoded sparse point cloud is input into the decoding normalization module to obtain the normalized sparse point cloud output by the decoding normalization module; the decoding normalization module is used to perform coordinate scaling on the decoded sparse point cloud. The normalized sparse point cloud is input into the structure initialization module to obtain the sparse point cloud location information output by the structure initialization module.

8. A point cloud compression method, characterized in that, include: Obtain target view parameters; The target viewpoint parameters are input into the point cloud compression model to obtain the target viewpoint reconstructed image output by the point cloud compression model; The point cloud compression model is trained based on the training method of the point cloud compression model according to any one of claims 1-7; Based on the reconstructed image from the target viewpoint and the preset camera parameters, the target depth map corresponding to the reconstructed image from the target viewpoint is determined; Based on the target depth map, a 3D consistency model is performed to obtain the target point cloud reconstruction data corresponding to the target viewpoint parameters.

9. A training device for a point cloud compression model, characterized in that, include: The first acquisition unit is used to acquire point cloud data; The encoding unit is used to input the point cloud data into the hierarchical encoder in the initial point cloud compression model to obtain the compressed encoded bitstream output by the hierarchical encoder; the hierarchical encoder is used to perform video encoding and geometric structure encoding on the point cloud data in parallel to obtain the compressed encoded bitstream after merging the bitstream. The decoding unit is used to input the compressed encoded bitstream into the decoder in the initial point cloud compression model to obtain the point cloud training data output by the decoder. The initial point cloud reconstruction model in the decoder is trained based on the point cloud training data to obtain the trained point cloud reconstruction model and point cloud compression model; the initial point cloud reconstruction model is constructed based on the three-dimensional Gaussian splashing algorithm.

10. A point cloud compression device, characterized in that, include: The second acquisition unit is used to acquire target view parameters; A reconstruction unit is used to input the target viewpoint parameters into a point cloud compression model to obtain a target viewpoint reconstructed image output by the point cloud compression model; the point cloud compression model is trained based on the training method of the point cloud compression model according to any one of claims 1-7. The determining unit is used to determine the target depth map corresponding to the target view reconstructed image based on the target view reconstructed image and preset camera parameters; The modeling unit is used to perform three-dimensional consistent modeling based on the target depth map to obtain target point cloud reconstruction data corresponding to the target viewpoint parameters.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the training method of the point cloud compression model as described in any one of claims 1 to 7, or the point cloud compression method as described in claim 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the point cloud compression model as described in any one of claims 1 to 7, or the point cloud compression method as described in claim 8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the point cloud compression model as described in any one of claims 1 to 7, or the point cloud compression method as described in claim 8.

Images