Point cloud attribute decoding method, point cloud attribute encoding method, encoder, decoder, and medium

Abstract

Embodiments of the present application relate to the technical field of point cloud encoding / decoding, and disclosed are a point cloud attribute decoding method, a point cloud attribute encoding method, an encoder, a decoder, and a medium. The method comprises: performing attribute decoding reconstruction on the basis of a code stream to obtain attributes of a reconstructed residual point cloud and attributes of a reconstructed underlying down-sampled point cloud; and concatenating, level by level, the attributes of the reconstructed underlying down-sampled point cloud and the attributes of the reconstructed residual point cloud to obtain attributes of a reconstructed point cloud.

Description

Point cloud attribute decoding, encoding method, codec and medium TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of point cloud coding, and particularly relate to a point cloud attribute decoding, encoding method, codec and medium. BACKGROUND

[0002] In a geometry-based point cloud compression (G-PCC) encoder framework, the geometry information and attribute information of a point cloud are encoded separately. After geometry encoding is completed, the geometry information is reconstructed, and the encoding of the attribute information depends on the reconstructed geometry information.

[0003] Since the amount of information of the attribute information is huge, how to improve the compression rate of the attribute information has become an important research direction in the field of point cloud coding.

[0004] SUMMARY

[0005] Embodiments of the present application provide a point cloud attribute decoding, encoding method, codec and medium. The technical solution is as follows:

[0006] On the one hand, the present application provides a point cloud attribute decoding method, which comprises:

[0007] Attribute decoding reconstruction is performed based on a code stream to obtain attributes of a reconstructed residual point cloud and attributes of a reconstructed bottom layer down-sampled point cloud;

[0008] The attributes of the reconstructed bottom layer down-sampled point cloud and the attributes of the reconstructed residual point cloud are spliced layer by layer to obtain attributes of a reconstructed point cloud.

[0009] On the other hand, the present application provides a point cloud attribute encoding method, which comprises:

[0010] The original point cloud is down-sampled multiple times to obtain multiple down-sampled point clouds of different levels;

[0011] Attribute feature extraction is performed on a residual point cloud between point clouds of adjacent levels to obtain residual point cloud attribute features;

[0012] The residual point cloud attribute features are encoded, and the bottom layer down-sampled point cloud is encoded.

[0013] On the other hand, the present application provides a point cloud attribute decoding device, which comprises:

[0014] A decoding unit is configured to perform attribute decoding reconstruction based on a code stream to obtain attributes of a reconstructed residual point cloud and attributes of a reconstructed bottom layer down-sampled point cloud;

[0015] The decoding unit is configured to concatenate the attributes of the reconstructed bottom-level down-sampled point cloud and the attributes of the reconstructed residual point cloud layer by layer to obtain the attributes of the reconstructed point cloud.

[0016] In another aspect, an embodiment of the present application provides a point cloud attribute encoding device, which comprises:

[0017] The encoding unit is configured to down-sample the original point cloud multiple times to obtain multiple down-sampled point clouds of different levels.

[0018] The encoding unit is configured to extract attribute features of the residual point cloud between the point clouds of adjacent levels to obtain residual point cloud attribute features.

[0019] The encoding unit is configured to encode the residual point cloud attribute features and encode the bottom-level down-sampled point cloud.

[0020] In another aspect, an embodiment of the present application provides a decoder, which comprises a memory and a processor, wherein the memory is configured to store a computer program running on the processor; and the processor is configured to execute a point cloud attribute decoding method as described in the above aspect when the computer program is running.

[0021] In another aspect, an embodiment of the present application provides an encoder, which comprises a memory and a processor, wherein the memory is configured to store a computer program running on the processor; and the processor is configured to execute a point cloud attribute encoding method as described in the above aspect when the computer program is running.

[0022] In another aspect, an embodiment of the present application provides a non-volatile computer readable storage medium storing a code stream, wherein the code stream is generated by using a point cloud encoding method of an encoder, or the code stream is decoded by using a point cloud decoding method of a decoder, wherein the point cloud encoding method comprises a point cloud attribute encoding method as described in the above aspect, and the point cloud decoding method comprises a point cloud attribute decoding method as described in the above aspect.

[0023] In another aspect, an embodiment of the present application provides a computer program product, which comprises computer instructions stored in a computer readable storage medium, wherein a processor acquires the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to implement a point cloud attribute encoding method as described in the above aspect, or a point cloud attribute decoding method.

[0024] By adopting the scheme provided in the embodiments of the present application, the encoding end determines the residual point cloud between adjacent level point clouds through point cloud downsampling and residual calculation, and extracts attribute features of the residual point cloud, so as to encode the attribute features of the residual point cloud and the bottom layer downsampled point cloud; after the attribute decoding of the decoding end, the attributes of the residual point cloud and the bottom layer downsampled point cloud are obtained, and the attributes of the reconstructed point cloud can be obtained through the layer-by-layer splicing manner. Since the residual calculation can eliminate the repeated or overlapped part of the point cloud features between different level point clouds, the feature extraction of the residual point cloud can reduce the redundancy, and help to improve the compression rate of the point cloud attribute. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 shows a flow block diagram of G-PCC encoding provided by the related art;

[0026] FIG. 2 shows a flow block diagram of G-PCC decoding provided by the related art;

[0027] FIG. 3 is a schematic diagram of a coding architecture according to an example embodiment of the present application;

[0028] FIG. 4 is a flow chart of a point cloud attribute encoding method according to an example embodiment of the present application;

[0029] FIG. 5 is a schematic diagram of an offset attention layer according to an example embodiment of the present application;

[0030] FIG. 6 is a schematic diagram of a normal information and attribute feature splicing process according to an example embodiment of the present application;

[0031] FIG. 7 is a structural schematic diagram of a feature extraction network of different levels according to an example embodiment of the present application;

[0032] FIG. 8 is a structural schematic diagram of an entropy model and a hypermodel according to an example embodiment of the present application;

[0033] FIG. 9 is a structural schematic diagram of a hyper-encoder, a hyper-decoder, a context model and a scaling factor determination model in a hypermodel according to an example embodiment of the present application;

[0034] FIG. 10 is a structural schematic diagram of a low-frequency sampling module according to an example embodiment of the present application;

[0035] FIG. 11 is a flow chart of a point cloud attribute encoding process according to an example embodiment of the present application;

[0036] FIG. 12 is an implementation schematic diagram of a point cloud attribute coding process according to an example embodiment of the present application;

[0037] FIG. 13 is a flow chart of a point cloud attribute decoding method according to an example embodiment of the present application;

[0038] FIG. 14 is a structural diagram of a feature restoration network of different levels according to an example embodiment of the present application;

[0039] FIG. 15 is a structural diagram of an attribute reconstruction network according to an example embodiment of the present application;

[0040] FIG. 16 is a flowchart of a point cloud attribute decoding process according to an example embodiment of the present application;

[0041] FIG. 17 is a structural block diagram of a point cloud attribute encoding apparatus according to an example embodiment of the present application;

[0042] FIG. 18 is a structural block diagram of a point cloud attribute decoding apparatus according to an example embodiment of the present application;

[0043] FIG. 19 is a structural block diagram of an encoder according to an example embodiment of the present application;

[0044] FIG. 20 is a structural block diagram of a decoder according to an example embodiment of the present application. DETAILED DESCRIPTION

[0045] For the purpose, technical solutions and advantages of the present application to be clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0046] “Multiple” mentioned herein refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0047] Referring to FIG. 1, a flowchart of G-PCC encoding provided by the related art is shown. As shown in the flowchart of G-PCC encoding in FIG. 1, the G-PCC encoding is applied to a point cloud encoder. For the point cloud data to be encoded, the point cloud data is first divided into multiple slices (point cloud slices) through slice division. In each slice, the geometry information of the point cloud and the attribute information corresponding to each point cloud are encoded separately.

[0048] In the geometry coding process, the coordinate transformation is performed on the geometry information, so that all the point clouds are contained in a bounding box, and then quantization is performed, which mainly plays a role in scaling. Due to quantization rounding, the geometry information of a part of the point clouds is the same, so whether to remove the duplicate points is determined based on the parameters, and the quantization and removal of duplicate points is also called the voxelization process. Then the bounding box is divided into an octree. In the octree-based geometry information coding process, the bounding box is equally divided into 8 sub-cubes, and the non-empty (containing points in the point cloud) sub-cubes are continuously divided into 8, until the division of the leaf node is stopped when the unit cube of 1x1x1 is obtained. The points in the leaf node are arithmetically coded to generate a binary geometry bit stream, i.e. a geometry code stream. In the triangle soup-based geometry information coding process, the octree division is also performed first, but unlike the octree-based geometry information coding, the triangle soup does not need to divide the point cloud to the unit cube with an edge length of 1x1x1, but to stop the division when the block (sub-block) edge length is W. Based on the distribution of the point cloud in each sub-block, the surface is obtained, and at most twelve vertices (intersection points) are generated by the surface and the twelve edges of the sub-block. The intersection points are arithmetically coded (surface fitting based on the intersection points) to generate a binary geometry bit stream, i.e. a geometry code stream. The intersection points are also used in the implementation of the geometry reconstruction process, and the reconstructed set information is used in the attribute coding of the point cloud.

[0049] In the attribute encoding process, after the geometry encoding completes the reconstruction of the geometry information, color conversion is performed to convert the color information (i.e., attribute information) from the RGB color space to the YUV color space. Then, the reconstructed geometry information is used to re-color the point cloud, so that the unencoded attribute information corresponds to the reconstructed geometry information. Attribute encoding is mainly performed on color information. In the color information encoding process, there are mainly two transformation methods. One is distance-based promotion transformation depending on level of detail (LOD) division. Currently, the division of LOD mainly includes two modes: distance-based division of LOD (mainly for Category 1 sequences) and fixed sampling rate-based division of LOD (mainly for Category 3 sequences). The other is direct region adaptive hierarchal transform (RAHT) transformation. In these two methods, the color information is converted from the spatial domain to the frequency domain, and high-frequency coefficients (AC) and low-frequency coefficients (DC) are obtained through transformation. The coefficients are quantized (i.e., quantized coefficients). Finally, the geometry encoding data obtained through octree division and surface fitting and the quantized coefficient processing attribute encoding data are combined through slice to encode the vertex coordinates (i.e., arithmetic encoding) of each block in turn, and a binary attribute bitstream, i.e., an attribute code stream, is generated.

[0050] Referring to FIG. 2, a flow block diagram of G-PCC decoding provided by the related technical solution is shown. As shown in the flow block diagram of G-PCC decoding in FIG. 2, applied to a point cloud decoder, for the obtained code stream, first, the geometry bitstream and the attribute bitstream in the code stream are independently decoded. In the decoding of the geometry bitstream, through arithmetic decoding-octree synthesis-surface fitting-reconstruction of geometry-inverse coordinate conversion, the geometry information of the point cloud is obtained. In the decoding of the attribute bitstream, through arithmetic decoding-inverse quantization-LOD-based promotion inverse transformation or RAHT-based inverse transformation-inverse color conversion, the attribute information of the point cloud is obtained, and based on the geometry information and the attribute information, the three-dimensional image model of the point cloud data to be encoded is restored.

[0051] The point cloud attribute encoding and decoding method provided by the embodiment of the present application adopts the coding and decoding framework as shown in FIG. 3. At the encoding end, the original geometry information 301 of the point cloud is encoded by the G-PCC geometry encoder 302 to obtain the geometry bitstream 303 and the reconstructed geometry information 304. The original attribute information 305 of the point cloud is subjected to attribute scaling and octree division, and then subjected to attribute encoding by the attribute encoder 306 to obtain the attribute bitstream 307. In the attribute encoding, the reconstructed geometry information 304 is used as auxiliary information to help attribute compression.

[0052] At the decoding end, the geometry bitstream 303 is decoded by the G-PCC geometry decoder 308 to obtain the reconstructed geometry information 304. The attribute bitstream 307 is decoded by the attribute decoder 309, and further processed by block merging and attribute inverse scaling to obtain the reconstructed attribute information 310. In this process, the reconstructed geometry information 304 is used as auxiliary information for attribute decoding.

[0053] Referring to FIG. 4, a flowchart of a point cloud attribute encoding method according to an example embodiment of the present application is shown. The method is used in an encoder, and can include the following steps:

[0054] At step 401, the original point cloud is down-sampled multiple times to obtain multiple down-sampled point clouds at different levels.

[0055] In some embodiments, the encoder down-samples the original point cloud based on the down-sampling of the previous level, i.e., the down-sampled point cloud obtained in the down-sampling process is gradually sparse.

[0056] In some embodiments, the encoder down-samples the original point cloud based on a down-sampling rule, which can include a frequency-based down-sampling rule, a distance-based down-sampling rule, etc. The specific down-sampling process under different down-sampling rules will be described in the following embodiments.

[0057] In some embodiments, the decoder down-samples the original point cloud a predetermined number of times, or dynamically down-samples the original point cloud based on the point cloud density of the original point cloud.

[0058] In an illustrative example, the decoder down-samples the original point cloud 3 times to obtain a first-level down-sampled point cloud, a second-level down-sampled point cloud, and a third-level down-sampled point cloud.

[0059] At step 402, attribute features of the residual point cloud between the point clouds at adjacent levels are extracted to obtain residual point cloud attribute features.

[0060] Although down-sampling the point cloud (multi-scale processing) can improve the encoding effect, it may introduce a certain degree of redundant information, especially in the feature transfer process between levels, the point cloud features of the down-sampled point clouds at different levels may overlap. Therefore, in order to reduce redundant information and reduce the overlap of point cloud features at different levels, in one possible implementation, the encoder does not directly extract features from the point cloud, but first determines the residual point cloud between the point clouds at adjacent levels, and then extracts attribute features from the residual point cloud to obtain residual point cloud attribute features.

[0061] The residual point cloud can be the residual point cloud between the original point cloud and the first-level down-sampled point cloud (obtained by down-sampling the original point cloud), or the residual point cloud between the down-sampled point clouds at adjacent levels.

[0062] Optionally, the residual point cloud is obtained by subtracting the point cloud, and the residual point cloud includes geometric information and attribute information of the points.

[0063] In a case where the residual point cloud is a residual point cloud between the original point cloud and the first layer down-sampled point cloud, the points in the residual point cloud belong to the original point cloud but not to the first layer down-sampled point cloud; in a case where the residual point cloud is a residual point cloud between down-sampled point clouds of adjacent levels, the points in the residual point cloud belong to the down-sampled point cloud of the previous level but not to the down-sampled point cloud of the current level.

[0064] In some embodiments, the residual point cloud includes residual point clouds of at least one level, that is, the encoder can determine residual point clouds of multiple levels based on the original point cloud and all down-sampled down-sampled point clouds, or determine residual point clouds of part of the levels based on the original point cloud and part of the down-sampled down-sampled point clouds.

[0065] In an illustrative example, when the original point cloud is down-sampled three times, the encoder can determine a residual point cloud between the original point cloud and the first layer down-sampled point cloud, a residual point cloud between the first layer down-sampled point cloud and the second layer down-sampled point cloud, and a residual point cloud between the second layer down-sampled point cloud and the third layer down-sampled point cloud; or, the encoder can determine a residual point cloud between the original point cloud and the first layer down-sampled point cloud, and a residual point cloud between the second layer down-sampled point cloud and the third layer down-sampled point cloud; or, the encoder only determines a residual point cloud between the original point cloud and the first layer down-sampled point cloud.

[0066] In some embodiments, the extracted attribute feature can be a color attribute feature, a reflectivity attribute feature, etc., which is not limited in the present embodiment.

[0067] Regarding the attribute feature extraction method, in a possible implementation, the encoder extracts attribute features of the residual point cloud by using a feature extraction network (neural network) to obtain residual point cloud attribute features.

[0068] Optionally, before extracting attribute features of the residual point cloud, the encoder scales the attribute values of the point cloud. The attribute scaling process is to make the attribute values be the maximum attribute value, so that the scaled attribute values are located between 0 and 1.

[0069] For example, for an 8-bit point cloud color attribute, the encoder divides the attribute values of the points in the point cloud by 255 to obtain the scaled point cloud color attribute.

[0070] Step 403, encode the residual point cloud attribute features and the bottom layer down-sampled point cloud.

[0071] After the attribute feature extraction is completed, the encoder encodes the residual point cloud attribute features and encodes the bottom layer down-sampled point cloud to obtain the attribute code stream. The bottom layer down-sampled point cloud is the down-sampled point cloud obtained through the last time of down-sampling.

[0072] In an illustrative example, when the original point cloud is down-sampled for 3 times and the residual point cloud attribute features of 3 levels of residual point clouds are extracted, the encoder encodes the residual point cloud attribute features of 3 levels of residual point clouds and encodes the attributes of the third layer down-sampled point cloud to obtain the attribute code stream.

[0073] In summary, in the embodiments of the present application, the encoder determines the residual point cloud between adjacent levels of point clouds through point cloud down-sampling and residual calculation, and extracts the attribute features of the residual point cloud, so as to encode the attribute features of the residual point cloud and the bottom layer down-sampled point cloud. After the attributes of the residual point cloud and the bottom layer down-sampled point cloud are reconstructed through attribute decoding at the subsequent decoding end, the attributes of the reconstructed point cloud can be obtained through the layer-by-layer splicing manner. Since the residual calculation can eliminate the repeated or overlapping parts of the point cloud features between different levels of point clouds, the feature extraction of the residual point cloud can reduce the redundancy and help to improve the compression rate of the point cloud attributes.

[0074] Encoding process of residual point cloud attribute features and bottom layer down-sampled point cloud

[0075] In some embodiments, for the residual point cloud, the encoder extracts the attribute features of the residual point cloud through the feature extraction network to obtain the residual point cloud attribute features, so as to write the residual point cloud attribute features into the attribute code stream.

[0076] For the bottom layer down-sampled point cloud, the encoder can obtain the bottom layer down-sampled point cloud attribute features through the feature extraction network, so as to write the bottom layer down-sampled point cloud attribute features into the attribute code stream. Alternatively, the encoder can directly encode the attributes of the bottom layer down-sampled point cloud without first performing attribute feature extraction.

[0077] In a possible implementation, the encoder can perform attribute encoding in the following manner.

[0078] Manner one, encode the residual point cloud attribute features; extract the attribute features of the bottom layer down-sampled point cloud to obtain the bottom layer down-sampled point cloud attribute features; and encode the bottom layer down-sampled point cloud attribute features.

[0079] Optionally, the feature extraction network used for the attribute feature extraction of the bottom layer down-sampled point cloud can be the same as or different from the feature extraction network used for the attribute feature extraction of the residual point cloud.

[0080] In some embodiments, the encoder respectively performs entropy encoding on the residual point cloud attribute feature and the bottom layer down-sampled point cloud attribute feature to obtain an attribute bitstream.

[0081] Optionally, the encoding processes of the residual point cloud attribute feature and the bottom layer down-sampled point cloud attribute feature can be independent of each other, or the encoding process of the residual point cloud attribute feature can refer to the encoding process of the bottom layer down-sampled point cloud attribute feature.

[0082] Method two, encoding the residual point cloud attribute feature; and encoding the bottom layer down-sampled point cloud attribute.

[0083] Since the bottom layer down-sampled point cloud is relatively sparse, and the attribute encoding efficiency of directly encoding the sparse point cloud is relatively high, and the reconstruction accuracy is relatively high, in a possible implementation, the encoder directly encodes the bottom layer down-sampled point cloud.

[0084] Optionally, the encoder performs G-PCC attribute encoding on the bottom layer down-sampled point cloud. For example, the encoder performs lifting transformation or RAHT transformation, quantization and entropy encoding on the bottom layer down-sampled point cloud to obtain an attribute bitstream of the bottom layer down-sampled point cloud. The attribute encoding process of the specific point cloud is not limited in the embodiments of the present application.

[0085] For the above-mentioned method one, in order to improve the attribute feature extraction quality, a respective feature extraction network can be set for each layer of residual point cloud and bottom layer down-sampled point cloud. Optionally, the network depths of different feature extraction networks are different.

[0086] In a possible implementation, the encoder extracts the attribute feature of the residual point cloud through the feature extraction network corresponding to the residual point cloud to obtain the residual point cloud attribute feature; and the encoder extracts the attribute feature of the bottom layer down-sampled point cloud through the feature extraction network corresponding to the bottom layer down-sampled point cloud to obtain the bottom layer down-sampled point cloud attribute feature.

[0087] The depth of the feature extraction network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud, that is, the feature extraction network corresponding to the lower layer residual point cloud can extract more deep attribute features.

[0088] In addition, since the bottom layer down-sampled point cloud is the basis for subsequent point cloud reconstruction, the depth of the feature extraction network corresponding to the bottom layer down-sampled point cloud is greater than the depth of the feature extraction network corresponding to the residual point cloud, so as to extract more deep attribute features of the bottom layer down-sampled point cloud.

[0089] In an illustrative example, when the original point cloud is downsampled three times and residual point cloud attribute features of three levels of residual point clouds are extracted, the depth of the feature extraction network corresponding to the third downsampled point cloud (i.e., the bottom downsampled point cloud) is greater than the depth of the feature extraction network corresponding to the third residual point cloud (the residual point cloud between the second and third downsampled point clouds) is greater than the depth of the feature extraction network corresponding to the second residual point cloud (the residual point cloud between the first and second downsampled point clouds) is greater than the depth of the feature extraction network corresponding to the first residual point cloud (the residual point cloud between the original point cloud and the first downsampled point cloud).

[0090] Regarding the structure of the feature extraction network, in one possible implementation, the feature extraction network includes convolutional layers, activation layers, and offset-attention layers.

[0091] The number of convolutional layers is positively correlated with the depth of the feature extraction network; that is, the deeper the feature extraction network, the more convolutional layers it has. In this embodiment, the number of convolutional layers in the feature extraction network corresponding to the bottom downsampled point cloud is greater than the number of convolutional layers in the feature extraction network corresponding to the residual point cloud; and the number of convolutional layers in the feature extraction network corresponding to the (i+1)th layer residual point cloud (the residual point cloud between the i-th layer downsampled point cloud and the (i+1)th layer downsampled point cloud) is greater than the number of convolutional layers in the feature extraction network corresponding to the i-th layer residual point cloud (the residual point cloud between the (i-1)th layer downsampled point cloud and the i-th layer downsampled point cloud).

[0092] When the network is deep, the self-attention layer cannot handle the problem of information loss. Considering the difference between the attention features and the input features, this embodiment uses offset attention to modify the features. The structure of the offset attention layer is illustrated in Figure 5.

[0093] Wherein, input F in First, through W q W k W v Perform a linear transformation to obtain Q, K, and V. Multiply Q and K by their transposes and then... After division (DIV), the result is subjected to softmax processing, and then the softmax result is multiplied by V to obtain the self-attention output F. sa Furthermore, for the input F in With self-attention output F sa The differences are weighted, and the result is used to calculate the weighted sum of the input F. in Determine the output F of the offset attention layer out .

[0094] The output of the offset attention layer can be represented as: Fout = OA(F in ) = γ(F in - F sa ) + F in (Q, K, V) = F in · (W q , W k , W v )

[0095] where OA is the offset attention mechanism, F in is the input of the offset attention, F sa is the output of the self-attention, γ is the weight coefficient, W q , W k , W v are shared learnable linear transformations, and d k is the dimension k of the key vector.

[0096] After the convolution layer, the offset attention mechanism is applied to calculate the correlation between different positions in the feature map, generate an attention matrix, and perform weighted processing on the feature map, which can effectively capture the mutual relationship between distant points and enhance the globality of the feature representation.

[0097] In some embodiments, in order to enhance the local spatial features of the point cloud, the convolution layer adopts sparse convolution.

[0098] In a possible implementation, in order to enhance the feature extraction quality, the input of the feature extraction network includes the geometric information and attribute information of the point cloud. The geometric information is the reconstructed geometric information of the encoder.

[0099] Correspondingly, the convolution layer in the feature extraction network adopts a three-dimensional convolution kernel for convolution operation to obtain a multi-channel feature map.

[0100] In some embodiments, in order to better capture the relationship and interaction between points, the feature extraction network further includes a feature fusion layer. When performing attribute feature extraction on the residual point cloud, the feature fusion layer is used to determine the normal information of the residual point cloud based on the geometric information of the residual point cloud, and to concatenate the normal information of the residual point cloud and the shallow attribute features of the residual point cloud to obtain the fusion features, wherein the shallow attribute features of the residual point cloud are output by the shallow convolution layer.

[0101] When extracting the attribute features of the bottom layer down-sampled point cloud, the feature fusion layer is configured to determine the normal information of the bottom layer down-sampled point cloud based on the geometric information of the bottom layer down-sampled point cloud, and splice the normal information of the bottom layer down-sampled point cloud and the shallow attribute features of the bottom layer down-sampled point cloud to obtain the fusion features. The shallow attribute features of the residual point cloud are output by the shallow convolution layer.

[0102] Optionally, the shallow attribute features are the attribute features output by the first layer convolution layer.

[0103] Illustratively, as shown in FIG. 6, the encoder calculates the normal information 62 of the point cloud based on the geometric information 61 of the point cloud, extracts the attribute information 63 through the convolution layer 64 to obtain the shallow attribute features 65, and splices the normal information 62 and the shallow attribute features 65 to obtain the fusion features 66.

[0104] In an illustrative example, when the attribute features of the residual point cloud of three levels are needed to be extracted, the feature extraction network corresponding to each level of residual point cloud and the bottom layer down-sampled point cloud is shown in FIG. 7.

[0105] The first feature extraction network 701 corresponding to the first layer residual point cloud includes one sparse convolution layer (SConv 3 3 ×128 s2↓), a feature fusion layer, and one offset attention layer (Offset-AT); the second feature extraction network 702 corresponding to the second layer residual point cloud includes two sparse convolution layers (SConv 3 3 ×128 and SConv 3 3 ×128 s2↓), and one offset attention layer (Offset-AT), and an activation layer (ReLU) is arranged after the feature fusion layer; the third feature extraction network 703 corresponding to the third layer residual point cloud includes three sparse convolution layers (SConv 3 3 ×128, SConv 3 3 ×128, and SConv 3 3 ×128 s2↓), and one offset attention layer (Offset-AT), and an activation layer (ReLU) is arranged between adjacent sparse convolution layers and after the feature fusion layer; the fourth feature extraction network 704 corresponding to the bottom layer down-sampled point cloud includes five sparse convolution layers (SConv 3 3 ×128, SConv 3 3 ×128, SConv 3 3 ×128 s2↓, SConv 3 3 ×128, and SConv 3 3and an offset attention layer (Offset-AT), and an activation layer (ReLU) is arranged between adjacent sparse convolution layers and after the feature fusion layer. The sparse convolution layer in each feature extraction network adopts a three-dimensional convolution kernel, and s2↓ represents 2 times down-sampling.

[0106] In the above embodiment, by configuring different depths of the feature extraction network for the residual point clouds of different levels and the bottom layer down-sampled point cloud, the feature extraction quality of the residual point clouds of lower levels and the bottom layer down-sampled point cloud is improved. Moreover, the feature extraction network adopts sparse convolution and offset attention mechanism, which can extract local features and global features of the point cloud, and improves the comprehensiveness of feature extraction.

[0107] In addition, the geometric information of the point cloud is input into the feature extraction network as auxiliary information together with the attribute information of the point cloud, which helps to improve the learning ability of the network on attribute correlation, and further improves the feature extraction quality of the point cloud.

[0108] In the above manner one, in order to further improve the coding efficiency, in a possible implementation, after the attribute feature extraction of the bottom layer down-sampled point cloud, the decoder determines a hyper-prior parameter based on the attribute feature of the bottom layer down-sampled point cloud, and the hyper-prior parameter is used for entropy coding.

[0109] In some embodiments, the hyper-prior parameter is determined by a hyper model (Hyperprior Model) based on the attribute feature of the bottom layer down-sampled point cloud. The hyper-prior parameter can include σ and μ.

[0110] In some embodiments, the hyper-prior parameter is transmitted to the entropy encoder of other layers, and correspondingly, the encoder encodes the attribute feature of the residual point cloud based on the hyper-prior parameter. Moreover, the encoder encodes the attribute feature of the bottom layer down-sampled point cloud based on the hyper-prior parameter.

[0111] In addition, in order to ensure the consistency of the entropy coding and decoding process at the encoding and decoding ends, the encoder encodes the hyper-prior parameter into the bitstream.

[0112] In some embodiments, the hyper model includes a hyper encoder (Hyper Encoder), a hyper decoder (Hyper Decoder) and a context model (Context Model).

[0113] The hyper encoder is used to encode the attribute feature of the bottom layer down-sampled point cloud to obtain hyper-prior information. The hyper decoder is used to decode the entropy coding and decoding result of the hyper-prior information to obtain hyper-prior decoding information. The context model is used to generate the hyper-prior parameter.

[0114] To improve the reconstruction quality while ensuring the compression efficiency, in some embodiments, a scaling factor determination model is further included in the hyper model. The scaling factor determination model is configured to determine adaptive scaling factors for the attribute features (including the residual point cloud attribute features and the underlying down-sampled point cloud attribute features) based on the hyper-prior information, so that the attribute features are adaptively quantized using the adaptive scaling factors subsequently.

[0115] In one possible implementation, the decoder determines the hyper-prior information based on the underlying down-sampled point cloud attribute features, and determines the adaptive scaling factors based on the hyper-prior information, so that the underlying down-sampled point cloud attribute features and the residual point cloud attribute features are adaptively quantized based on the adaptive scaling factors.

[0116] Further, the encoder encodes the adaptively quantized residual point cloud attribute features based on the hyper-prior parameters, and encodes the adaptively quantized underlying down-sampled point cloud attribute features based on the hyper-prior parameters.

[0117] Optionally, the decoder encodes the underlying down-sampled point cloud attribute features through the hyper-encoder to obtain the hyper-prior information.

[0118] Optionally, the decoder determines the adaptive scaling factors based on the hyper-prior information through the scaling factor determination model.

[0119] Optionally, the adaptive scaling factors are used for hard quantization of the attribute features. For a feature y and an adaptive scaling factor Δ, the hard quantization process can be represented as wherein, represents the floor function.

[0120] Illustratively, as shown in FIG. 8, the hyper model 810 includes a hyper-encoder 811, a hyper-decoder 812, a context model 813, and a scaling factor determination model 814.

[0121] The entropy model 820 corresponding to the underlying down-sampled point cloud quantizes the underlying down-sampled point cloud attribute features y L The hyper-encoder 811 of the hyper model 810 obtains the hyper-prior information z L The hyper-prior information z L After simple quantization (Q), the quantized hyper-prior information The scaling factor determination model 814 determines the adaptive scaling factors Δ based on the quantized hyper-prior information The adaptive scaling factors Δ are generated, which will be provided to the entropy model 820 corresponding to the underlying down-sampled point cloud, and the entropy models 830 corresponding to the residual point clouds of each layer.

[0122] The decomposed entropy model 815 based on the Gaussian distribution quantizes the quantized hyper-prior information After entropy coding (AE and AD), the hyper-prior information entropy coding result is obtained The hyper-decoder 812 takes the hyper-prior information entropy coding result as input After decoding, the hyper-prior decoding result is obtained The context model 813 is based on the hyper-prior decoding result and after adaptive quantization processing Q Δ The adaptive quantization bottom layer down-sampled feature is obtained The hyper-prior parameters σ and μ are generated and provided to the entropy model 820 and the entropy model 830

[0123] The entropy model 820 corresponding to the bottom layer down-sampled point cloud first uses the adaptive scaling factor Δ to perform adaptive quantization processing Q on the bottom layer down-sampled point cloud attribute feature y L The adaptive quantization processing Q is performed Δ Then, the adaptive quantization processing result is entropy encoded and decoded based on the hyper-prior parameters σ and μ to obtain the output

[0124] The entropy model 830 corresponding to the residual point cloud first uses the adaptive scaling factor Δ to perform adaptive quantization processing Q on the residual point cloud attribute feature y l The adaptive quantization processing Q is performed Δ Then, the adaptive quantization processing result is entropy encoded and decoded based on the hyper-prior parameters σ and μ to obtain the output

[0125] As shown in FIG. 9, the input of the hyper-encoder 811 is the bottom layer down-sampled point cloud attribute feature y L , and the output is the hyper-prior information z L . The hyper-encoder 811 sequentially includes a first convolutional layer (SConv 3 3 ×128), a first activation layer (ReLU), a second convolutional layer (SConv 3 3 ×128), a third convolutional layer (SConv 3 3 ×128 s2↓), a second activation layer (ReLU), a fourth convolutional layer (SConv 3 3 ×128), a third activation layer (ReLU), and a fifth convolutional layer (SConv 3 3 ×128 s2↓). Among them, s2↓ represents two times down-sampling.

[0126] The input of the hyper-decoder 812 is the entropy coding result of the quantized hyper-prior information based on the factorized entropy model of the Gaussian distribution The output is the hyper-prior decoding result The hyper-decoder 812 sequentially comprises a first convolutional layer (SConv 3 3 ×128 s2↑), a first activation layer (ReLU), a second convolutional layer (SConv 3 3 ×128), a second activation layer (ReLU), a third convolutional layer (SConv 3 3 ×256 s2↑), a fourth convolutional layer (SConv 3 3 ×256), a third activation layer (ReLU), a fifth convolutional layer (SConv 3 3 ×256). Wherein, s2↑ represents two times up-sampling.

[0127] The input of the context model 813 is the hyper-prior encoding result and the adaptive quantization result of the bottom layer down-sampling feature The output is the hyper-prior parameters σ and μ. The context model 813 sequentially comprises a first convolutional layer (Mask SConv 5 3 ×256), a splicing layer (concat), a second convolutional layer (SConv 1 3 ×256), a first activation layer (ReLU), a third convolutional layer (SConv 1 3 ×256), a second activation layer (ReLU), and a fourth convolutional layer (SConv 1 3 ×256).

[0128] The input of the scaling factor determination model 814 is the quantized hyper-prior information The output is the adaptive scaling factor Δ. The scaling factor determination model 814 sequentially comprises a first convolutional layer (SConv 3 3 ×128), a first activation layer (ReLU), a second convolutional layer (SConv 3 3 ×128), and a second activation layer (sigmoid).

[0129] It should be noted that the above process of transmitting the hyper-prior parameters and the adaptive scaling factor is optional. The encoder can also not determine the hyper-prior parameters, but independently encode each layer by itself. Correspondingly, the encoder does not need to encode the hyper-prior parameters into the code stream, which is not limited in the present embodiment.

[0130] In the above embodiment, the hyper-model is introduced for entropy coding in the encoding process, and the hyper-prior parameters determined based on the bottom layer down-sampling point cloud attribute features are transmitted to other layers for encoding the residual point cloud attribute features, which helps to improve the encoding efficiency.

[0131] In addition, a scaling factor determination model is additionally added, an adaptive scaling factor of the attribute feature is determined by using the scaling factor determination model, and the attribute feature is adaptively quantized by using the adaptive scaling factor, so that the reconstruction quality is improved while the compression efficiency is ensured.

[0132] A complete point cloud can contain millions of points, and the common compression of all attributes of the complete point cloud consumes a huge amount of calculation and storage of the network. Therefore, in the embodiment, the encoder divides and blocks the point cloud in the same way as the octree division in G-PCC, so as to improve the access efficiency of the point cloud data and provide a structured input for feature extraction and coding.

[0133] In some embodiments, the encoder performs octree division on the residual point cloud and the underlying down-sampled point cloud to obtain hierarchical residual sub-point clouds and hierarchical down-sampled sub-point clouds.

[0134] Correspondingly, the encoder extracts attribute features from the hierarchical residual sub-point clouds to obtain residual point cloud attribute features, and extracts attribute features from the hierarchical down-sampled sub-point clouds to obtain underlying down-sampled point cloud attribute features.

[0135] In a possible implementation, the encoder can extract attribute features from different hierarchical residual sub-point clouds respectively, and splice the residual sub-point cloud attribute features of the different hierarchical residual sub-point clouds to obtain the residual point cloud attribute features.

[0136] In a possible implementation, the encoder can extract attribute features from different hierarchical down-sampled sub-point clouds respectively, and splice the down-sampled sub-point cloud attribute features of the different hierarchical down-sampled sub-point clouds to obtain the underlying down-sampled point cloud attribute features.

[0137] The octree division on the residual point cloud and the underlying down-sampled point cloud can adopt any of the following strategies:

[0138] Strategy 1, the residual point cloud and the underlying down-sampled point cloud are octree-divided according to the same octree depth.

[0139] Optionally, after the octree division, the octree layers of the octrees corresponding to different residual point clouds and underlying down-sampled point clouds are the same, and the sizes of the octree end leaf nodes are the same.

[0140] For example, the encoder octree-divides the residual point cloud and the underlying down-sampled point cloud according to an octree depth of 3.

[0141] Strategy 2, the residual point cloud and the underlying down-sampled point cloud are octree-divided according to different octree depths, and different residual point clouds are octree-divided according to different octree depths.

[0142] In some embodiments, different from stopping the division in G-PCC until the size of the leaf node is 1x1x1, the encoder adopts different depths of octree for different down-sampled point clouds in attribute encoding.

[0143] Since the importance of the lower residual point cloud and the bottom down-sampled point cloud to the subsequent point cloud reconstruction is higher (i.e. the impact on the point cloud reconstruction effect is greater), in a possible implementation, the octree depth corresponding to the bottom down-sampled point cloud is smaller than the octree depth corresponding to the residual point cloud, and the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

[0144] Optionally, the octree depths corresponding to the residual point clouds of different levels are different, or the octree depths corresponding to the residual point clouds of part of the levels are the same, and the octree depths corresponding to the residual point clouds of part of the levels are different.

[0145] In an illustrative example, when there are three levels of residual point clouds, the octree depths of the octrees corresponding to the first and second levels of residual point clouds are 2, the octree depth of the octree corresponding to the third level of residual point cloud is 3, and the octree depth of the octree corresponding to the bottom down-sampled point cloud is 4.

[0146] In some embodiments, the size of the leaf node in the octree obtained by the division is 8x8x8.

[0147] In this embodiment, by performing octree division on the point cloud, a structured input is provided for feature extraction and encoding, and the access efficiency of the point cloud data is improved. Moreover, the encoder adopts different depths of octree division for the residual point clouds of different levels and the bottom down-sampled point cloud, so as to ensure the reconstruction quality at the decoding end after subsequent feature extraction.

[0148] For the above-mentioned method two, in order to improve the attribute feature extraction quality, a respective feature extraction network can be set for each level of residual point cloud. Optionally, the network depths of different feature extraction networks are different.

[0149] In a possible implementation, the encoder performs attribute feature extraction on the residual point cloud through the feature extraction network corresponding to the residual point cloud, to obtain the attribute feature of the residual point cloud.

[0150] In this embodiment, the depth of the feature extraction network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud, i.e. the feature extraction network corresponding to the lower residual point cloud can extract more deep-level attribute features.

[0151] In an illustrative example, when the original point cloud is down-sampled 3 times and the residual point cloud attribute features of 3 levels of residual point clouds are extracted, the depth of the feature extraction network corresponding to the third level of residual point cloud (the residual point cloud between the second level of down-sampled point cloud and the third level of down-sampled point cloud) > the depth of the feature extraction network corresponding to the second level of residual point cloud (the residual point cloud between the first level of down-sampled point cloud and the second level of down-sampled point cloud) > the depth of the feature extraction network corresponding to the first level of residual point cloud (the residual point cloud between the original point cloud and the first level of down-sampled point cloud).

[0152] Regarding the structure of the feature extraction network, in a possible implementation, the feature extraction network includes a convolution layer, an activation layer, and an offset-attention layer.

[0153] The number of convolution layers is positively correlated with the depth of the feature extraction network, that is, the deeper the feature extraction network, the more the number of convolution layers. In the embodiments of the present application, the number of convolution layers in the feature extraction network corresponding to the bottom level of down-sampled point cloud is greater than the number of convolution layers in the feature extraction network corresponding to the residual point cloud; and the number of convolution layers in the feature extraction network corresponding to the i+1 level of residual point cloud (the residual point cloud between the i level of down-sampled point cloud and the i+1 level of down-sampled point cloud) is greater than the number of convolution layers in the feature extraction network corresponding to the i level of residual point cloud (the residual point cloud between the i-1 level of down-sampled point cloud and the i level of down-sampled point cloud).

[0154] When the network depth is deep, the self-attention layer cannot handle the problem of information loss. Considering the difference between the attention features and the input features, the offset attention is used in the embodiments to modify the features.

[0155] After the convolution layer, the offset attention mechanism is applied to calculate the correlation between different positions in the feature map, generate an attention matrix, and perform weighted processing on the feature map, which can effectively capture the mutual relationship between distant points and enhance the globality of feature representation.

[0156] In some embodiments, in order to enhance the local spatial features of the point cloud, the convolution layer adopts sparse convolution.

[0157] In a possible implementation, in order to enhance the feature extraction quality, the input of the feature extraction network includes the geometric information and attribute information of the point cloud. The geometric information is the reconstructed geometric information of the encoder.

[0158] Correspondingly, the convolution layer in the feature extraction network adopts a three-dimensional convolution kernel for convolution operation to obtain a multi-channel feature map.

[0159] In an illustrative example, when the residual point cloud attribute features of the 3-level residual point cloud are to be extracted, the first feature extraction network corresponding to the first-level residual point cloud comprises one sparse convolution layer and one offset attention layer; the second feature extraction network corresponding to the second-level residual point cloud comprises two sparse convolution layers and one offset attention layer, and an activation layer is arranged between the two sparse convolution layers; and the third feature extraction network corresponding to the third-level residual point cloud comprises three sparse convolution layers and one offset attention layer, and an activation layer is arranged between adjacent sparse convolution layers. In each feature extraction network, the sparse convolution layer uses a three-dimensional convolution kernel.

[0160] In some embodiments, the encoder performs octree division on the residual point cloud and the bottom layer down-sampled point cloud to obtain hierarchical residual sub-point clouds and hierarchical down-sampled sub-point clouds.

[0161] Correspondingly, the encoder extracts attribute features from the hierarchical residual sub-point clouds to obtain residual point cloud attribute features.

[0162] In a possible implementation, the encoder can extract attribute features from different hierarchical residual sub-point clouds respectively, and concatenate the residual sub-point cloud attribute features of different hierarchical residual sub-point clouds to obtain residual point cloud attribute features.

[0163] The octree division on the residual point cloud can use any of the following strategies:

[0164] Strategy 1: Different residual point clouds are divided by the same octree depth.

[0165] Optionally, after octree division, the octrees corresponding to different residual point clouds have the same number of octree layers, and the sizes of the end leaf nodes of the octrees are the same.

[0166] For example, the encoder divides the residual point clouds by octree depth 3.

[0167] Strategy 2: Different residual point clouds are divided by different octree depths.

[0168] Since the lower the level of the residual point cloud, the more important it is for subsequent point cloud reconstruction (i.e., the greater the impact on point cloud reconstruction), in a possible implementation, the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

[0169] Optionally, the octree depths corresponding to different levels of residual point clouds are different, or the octree depths corresponding to part of the levels of residual point clouds are the same, and the octree depths corresponding to part of the levels of residual point clouds are different.

[0170] In an illustrative example, when there are three layers of residual point clouds, the first and second layers of residual point clouds correspond to an octree depth of 2 of the octree, and the third layer of residual point clouds corresponds to an octree depth of 3 of the octree.

[0171] In some embodiments, the size of the leaf nodes in the divided octree is 8x8x8.

[0172] In some embodiments, the encoder can perform point cloud downsampling in the following manner:

[0173] Manner one: the original point cloud is downsampled multiple times by a low-frequency sampling module to obtain multiple downsampled point clouds at different levels, and the low-frequency sampling module is used to extract and filter low-frequency regions in the point cloud.

[0174] The low-frequency sampling module (LowFreqSp Module) is a pre-trained module used to selectively sample points from the point cloud, filter low-frequency points in the point cloud, and retain high-frequency points in the point cloud. Among them, the low-frequency points are regions with small changes or smooth regions in the point cloud (such as regions with the same color on an object), and the high-frequency points are regions with geometric or color mutations in the point cloud (such as the outline region of an object).

[0175] Correspondingly, the residual point cloud calculated by the residual calculation is the filtered low-frequency point cloud.

[0176] Correspondingly, when the original point cloud is downsampled multiple times by the low-frequency sampling module, the lower the layer of the downsampled point cloud, the higher the frequency, that is, the more prominent features of the original point cloud can be represented.

[0177] In some embodiments, a unified low-frequency sampling module is used for downsampling at each level.

[0178] In a possible implementation, in order to improve the comprehensiveness of low-frequency sampling (to identify regions with geometric and color mutations), the input of the low-frequency sampling module includes geometric information and attribute information of the point cloud. Among them, the geometric information is reconstructed geometric information.

[0179] Structure of the low-frequency sampling module

[0180] In a possible design, as shown in FIG. 10, the low-frequency sampling module 1000 includes a Hamming window processing submodule 1001, a Fast Fourier Transform (FFT) submodule 1002, a frequency domain sampling network 1003, and an Inverse Fast Fourier Transform (IFFT) submodule 1004.

[0181] The frequency domain sampling network 1003 is configured to extract and filter low-frequency frequency domain data. As shown in FIG. 10, the input of the frequency domain sampling network 1003 is the original geometric information g orig and the original attribute information c orig , and the output is the high-frequency geometric information g high and the high-frequency attribute information c high .

[0182] Optionally, the input point cloud is preprocessed by a Hamming window processing submodule (to reduce spectral leakage). The Hamming window preprocessing process can be represented as:

[0183] The coefficient ω(n) of the Hamming window is calculated at point n, N is the length of the window function, which is usually the same as the length of the input signal, a m is a coefficient that determines the shape of the window function, which can be optimized according to the characteristics of the signal, and M is a coefficient that determines the shape complexity. The window function is applied to each component of the input signal, i.e., the geometry and the attribute: g′ orig = ω(n)·g orig , c′ orig = ω(n)·c orig

[0184] where g orig is the original geometric information and c orig is the original attribute information.

[0185] After Hamming window preprocessing, the encoder performs FFT transformation to convert the geometry and attribute information into the frequency domain. Then, a neural network-based high-pass filter is used to selectively sample the frequency domain signal. The filtered frequency domain data is converted back to the spatial domain by inverse fast Fourier transform to recover and reconstruct the geometry and attribute.

[0186] In some embodiments, the frequency domain sampling network 1003 includes MultiLayer Perceptron (MLP) layers, Global Avg pooling layers, self-attention layers, convolution layers (Conv, including Conv C 3 ×d / 2, and Conv C 3 ×d), and Max pooling layers, where d is the number of dimensions.

[0187] In a possible implementation, to realize adaptive sampling of the high-frequency signal, the frequency domain sampling network 1003 is a neural network based scoring model (NNSM). The frequency domain sampling network outputs a weight vector Ω, and then multiplies the weight vector with the frequency coefficients to enhance the high-frequency coefficients and suppress the low-frequency coefficients, where the weight vector is used to represent the importance of each frequency coefficient. The process can be represented as: v" orig = Ω · v' orig

[0188] where v' orig includes g' orig and c' orig .

[0189] In addition, the low-frequency sampling module 1003 also introduces a threshold value φ to clip the frequency coefficients after weight processing. The threshold value φ can be a percentile of the weight vector, that is, φ = percentile (Ω, q), and q is the percentile. The process of clip processing based on the threshold value φ can be represented as:

[0190] After the above processing, important high-frequency features can be retained and low-frequency features can be removed, and effective data compression can be realized without losing too much information.

[0191] Method two: multiple farthest point sampling is performed on the original point cloud to obtain multiple down-sampled point clouds at different levels.

[0192] Farthest point sampling (FPS) is a technology that realizes uniform sampling of point cloud data by iteratively selecting the farthest point, which can effectively retain the spatial distribution characteristics of the point cloud, and maintain the geometry and attribute information of the point cloud on the basis of uniform sampling.

[0193] Of course, in addition to the above down-sampling scheme, the encoder can also use other optional down-sampling schemes, which are not limited in the present embodiment.

[0194] In the above embodiments, when determining the residual point cloud, in a possible implementation, the encoder first determines the residual point cloud layer number, and then determines the residual point cloud between point clouds at adjacent levels based on the residual point cloud layer number.

[0195] In some embodiments, the residual point cloud layer number can be a fixed value or a dynamic value.

[0196] In a possible implementation, when the residual point cloud layer number is a dynamic value, the residual point cloud layer number can be determined according to a preset index.

[0197] Optionally, the preset index can be a code rate.

[0198] In some embodiments, when the code rate is low, the encoder can determine and encode only part of the lower residual point clouds; when the code rate is high, the encoder can determine and encode all the lower residual point clouds. Wherein, with the increase of the code rate, the encoding of one or more upper residual point clouds can be increased.

[0199] In a possible implementation, the encoding end is provided with a corresponding relationship between the code rate and the number of residual point clouds, so as to determine the number of residual point clouds currently used based on the corresponding relationship and the current code rate.

[0200] Further, in the case that m times of downsampling obtain m layers of downsampled point clouds and the number of residual point clouds is n layers, the encoder determines the mth to the m-n+1th layers of residual point clouds, and n is less than or equal to m.

[0201] Wherein, the first layer of residual point clouds is the residual point cloud between the original point cloud and the first layer of downsampled point clouds, the ith layer of residual point clouds is the residual point cloud between the ith layer of downsampled point clouds and the (i-1)th layer of downsampled point clouds, the first layer of downsampled point clouds is obtained by downsampling the original point cloud, the ith layer of downsampled point clouds is obtained by downsampling the (i-1)th layer of downsampled point clouds, and i is an integer greater than or equal to 2.

[0202] In an illustrative example, when three times of downsampling obtain 3 layers of downsampled point sources and the number of residual point clouds is 3 layers, the encoder determines the first layer of residual point clouds, the second layer of residual point clouds and the third layer of residual point clouds, i.e. determines all the residual point clouds; when the number of residual point clouds is 2 layers, the encoder determines the second layer of residual point clouds and the third layer of residual point clouds, i.e. determines part of the residual point clouds; when the number of residual point clouds is 1 layer, the encoder determines the third layer of residual point clouds, i.e. determines part of the residual point clouds.

[0203] It should be noted that in the case of extremely poor code rate quality, the encoder can also only encode the bottom layer of downsampled point clouds without encoding the residual point clouds to ensure the most basic display requirements.

[0204] In this embodiment, by introducing a scalable encoding strategy, the number of residual point clouds is dynamically determined to realize the encoding of part or all of the residual point clouds. For example, the number of residual point clouds can be adjusted according to the code rate to adapt to different transmission and display requirements.

[0205] In combination with the above various embodiments, in an illustrative example, as shown in FIG. 11, the point cloud attribute encoding process can include the following steps.

[0206] Step 1101, multiple down-sampling of the original point cloud is performed by a low-frequency sampling module to obtain multiple down-sampled point clouds at different levels. The low-frequency sampling module is used to extract and filter low-frequency regions in the point cloud.

[0207] As shown in FIG. 12, the encoder samples the original point cloud 1201 by the low-frequency sampling module to obtain the first layer down-sampled point cloud 1202; samples the first layer down-sampled point cloud 1202 by the low-frequency sampling module to obtain the second layer down-sampled point cloud 1203; and samples the second layer down-sampled point cloud 1203 by the low-frequency sampling module to obtain the third layer down-sampled point cloud 1204.

[0208] Step 1102, the residual point cloud between the point clouds at adjacent levels is determined.

[0209] In some embodiments, the encoder determines the number of residual point cloud layers that need to be encoded according to the code rate.

[0210] As shown in FIG. 12, the encoder determines the first residual point cloud 1205 between the original point cloud 1201 and the first layer down-sampled point cloud 1202; determines the second residual point cloud 2106 between the first layer down-sampled point cloud 1202 and the second layer down-sampled point cloud 1203; and determines the third residual point cloud 1207 between the second layer down-sampled point cloud 1203 and the third layer down-sampled point cloud 1204.

[0211] Step 1103, the residual point cloud and the underlying down-sampled point cloud are octree-divided to obtain hierarchical residual sub-point clouds and hierarchical down-sampled point clouds.

[0212] In some embodiments, the encoder adopts different octree depths to perform octree division on the residual point cloud and the underlying down-sampled point cloud.

[0213] As shown in FIG. 12, the encoder performs octree division on the first residual point cloud 1205, the second residual point cloud 1206, the third residual point cloud 1207, and the third layer down-sampled point cloud 1204, respectively.

[0214] Step 1104, attribute feature extraction is performed on the hierarchical down-sampled point cloud by the feature extraction network corresponding to the underlying down-sampled point cloud to obtain the underlying down-sampled point cloud attribute feature.

[0215] In some embodiments, the depth of the feature extraction network corresponding to the underlying down-sampled point cloud is greater than the depth of the feature extraction network corresponding to the residual point cloud.

[0216] As shown in FIG. 12, the encoder adopts the fourth feature extraction network 1211 to perform attribute feature extraction on the hierarchical down-sampled point cloud obtained by dividing the third layer down-sampled point cloud 1204 to obtain the underlying down-sampled point cloud attribute feature.

[0217] Step 1105, encode the bottom layer down-sampled point cloud attribute features.

[0218] As shown in FIG. 9, the encoder entropy encodes the bottom layer down-sampled point cloud attribute features through the entropy model 1212.

[0219] Step 1106, determine the hyper-prior parameters based on the bottom layer down-sampled point cloud attribute features, the hyper-prior parameters are used for entropy encoding, and the hyper-prior parameters are encoded into the bitstream.

[0220] As shown in FIG. 12, the encoder determines the hyper-prior parameters through the hyper model 1218 based on the bottom layer down-sampled point cloud attribute features, and transmits the hyper-prior parameters to the entropy model 1212 of other levels.

[0221] Step 1107, extract attribute features of the hierarchical residual sub-point cloud through the feature extraction network corresponding to the residual point cloud, to obtain residual point cloud attribute features.

[0222] In some embodiments, the depths of the feature extraction networks corresponding to the residual point clouds of different levels are different.

[0223] As shown in FIG. 12, the encoder extracts attribute features of the hierarchical residual sub-point cloud obtained by dividing the first residual point cloud 1205 through the first feature extraction network 1208, to obtain first residual point cloud attribute features; extracts attribute features of the hierarchical residual sub-point cloud obtained by dividing the second residual point cloud 1206 through the second feature extraction network 1209, to obtain second residual point cloud attribute features; and extracts attribute features of the hierarchical residual sub-point cloud obtained by dividing the third residual point cloud 1207 through the third feature extraction network 1210, to obtain third residual point cloud attribute features.

[0224] Step 1108, encode the residual point cloud attribute features based on the hyper-prior parameters.

[0225] As shown in FIG. 12, the entropy model 1212 corresponding to each level entropy encodes the respective residual point cloud attribute features based on the hyper-prior parameters.

[0226] Please refer to FIG. 13, which shows a flow chart of a point cloud attribute decoding method provided by an example embodiment of the present application. The method is used for a decoder, and the method can include the following steps:

[0227] Step 1301, perform attribute decoding reconstruction based on the bitstream, to obtain attributes of the reconstructed residual point cloud and attributes of the reconstructed bottom layer down-sampled point cloud.

[0228] As described in the above-mentioned embodiment of the encoding side, the decoding side receives the residual point cloud attribute features of the residual point cloud and the bottom layer down-sampled point cloud in the code stream, and thus the decoder parses the code stream and reconstructs the attributes to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampled point cloud.

[0229] The number of layers of the reconstructed residual point cloud is consistent with the number of layers of the residual point cloud encoded by the encoding side.

[0230] Optionally, the point cloud attribute can include a color attribute or a reflectivity attribute, etc., which is not limited in the embodiment.

[0231] In step 1302, the attributes of the reconstructed bottom layer down-sampled point cloud and the attributes of the reconstructed residual point cloud are spliced layer by layer to obtain the attributes of the reconstructed point cloud.

[0232] Since the residual point cloud is the residual between the down-sampled point clouds of adjacent layers, and the bottom layer down-sampled point cloud is the point cloud obtained by the last time of down-sampling, the decoder splices the reconstructed residual point cloud to the reconstructed bottom layer down-sampled point cloud layer by layer to obtain the reconstructed point cloud.

[0233] In an illustrative example, when there are three layers of reconstructed residual point clouds and the reconstructed bottom layer down-sampled point cloud is the third layer down-sampled point cloud , the reconstructed point cloud can be represented as:

[0234] In summary, in the embodiment of the application, the encoding side determines the residual point cloud between the point clouds of adjacent layers by means of point cloud down-sampling and residual calculation, and extracts the attribute features of the residual point cloud, so as to encode the residual point cloud attribute features and the bottom layer down-sampled point cloud. After the decoding side reconstructs the attributes of the residual point cloud and the bottom layer down-sampled point cloud through attribute decoding, the attributes of the reconstructed point cloud can be obtained through layer-by-layer splicing. Since the residual calculation can eliminate the repeated or overlapped parts of the point cloud features between different layers of point clouds, the feature extraction of the residual point cloud can reduce the redundancy and help to improve the compression rate of the point cloud attribute.

[0235] Attribute decoding and reconstruction process

[0236] Corresponding to the process of the encoding side, the attribute decoding and reconstruction of the decoder includes the following two ways:

[0237] In the first way, the code stream is decoded and the features are recovered to obtain the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features; and the attributes are reconstructed based on the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampled point cloud.

[0238] The decoding reconstruction manner corresponds to the encoding manner at the encoding end. In this manner, both the residual point cloud and the bottom layer down-sampled point cloud are subjected to feature extraction, and the code stream contains the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features.

[0239] The decoder first decodes and recovers features from the code stream to obtain the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features, and then reconstructs the residual point cloud based on the residual point cloud attribute features and reconstructs the bottom layer down-sampled point cloud based on the bottom layer down-sampled point cloud attribute features.

[0240] The feature recovery and the point cloud reconstruction are mirror processes of the feature extraction at the encoding end.

[0241] In a possible implementation, the decoder decodes the code stream and recovers features through a feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features.

[0242] The decoder recovers features through a feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampled point cloud attribute features.

[0243] Optionally, the feature recovery network and the attribute reconstruction network are mirror networks of the feature extraction network at the encoding end.

[0244] In some embodiments, in the case where the same feature extraction network is used at the encoding end to extract attribute features for the residual point cloud and the bottom layer down-sampled point cloud, the decoder also recovers attribute features through the same feature recovery network.

[0245] In other embodiments, in the case where different feature extraction networks are used at the encoding end to extract attribute features for the residual point cloud and the bottom layer down-sampled point cloud, the decoder also needs to recover attribute features through different feature recovery networks.

[0246] Optionally, the different feature recovery networks have different network depths.

[0247] In a possible implementation, the decoder recovers features through the feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute features.

[0248] The decoder recovers features through the feature recovery network corresponding to the bottom layer down-sampled point cloud to obtain the bottom layer down-sampled point cloud attribute features.

[0249] In this case, since a deeper network is used at the encoding end to extract features for the lower layer residual point cloud, a deeper network needs to be used at the decoding end to recover features, i.e., the depth of the feature recovery network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud.

[0250] Since a deeper network is used to extract features of the bottom layer down-sampled point cloud at the encoding end, a deeper network is needed to recover the features of the down-sampled point cloud at the decoding end, that is, the depth of the feature recovery network corresponding to the bottom layer down-sampled point cloud is greater than the depth of the feature recovery network corresponding to the residual point cloud.

[0251] In an illustrative example, when the residual point cloud attribute features of three-level residual point clouds are written in the code stream, the depth of the feature recovery network corresponding to the third layer down-sampled point cloud (i.e., the bottom layer down-sampled point cloud) > the depth of the feature recovery network corresponding to the third layer residual point cloud (the residual point cloud between the second layer down-sampled point cloud and the third layer down-sampled point cloud) > the depth of the feature recovery network corresponding to the second layer residual point cloud (the residual point cloud between the first layer down-sampled point cloud and the second layer down-sampled point cloud) > the depth of the feature recovery network corresponding to the first layer residual point cloud (the residual point cloud between the original point cloud and the first layer down-sampled point cloud).

[0252] Regarding the structure of the feature recovery network, in a possible implementation, the feature recovery network includes a deconvolution layer and an activation layer. The number of deconvolution layers is positively correlated with the depth of the feature recovery network.

[0253] The number of deconvolution layers is positively correlated with the depth of the feature recovery network, that is, the deeper the feature recovery network, the more the number of deconvolution layers. In the embodiment of the present application, the number of deconvolution layers in the feature recovery network corresponding to the bottom layer down-sampled point cloud is greater than the number of deconvolution layers in the feature recovery network corresponding to the residual point cloud; and the number of deconvolution layers in the feature recovery network corresponding to the i+1 layer residual point cloud (the residual point cloud between the i layer down-sampled point cloud and the i+1 layer down-sampled point cloud) is greater than the number of deconvolution layers in the feature recovery network corresponding to the i layer residual point cloud (the residual point cloud between the i-1 layer down-sampled point cloud and the i layer down-sampled point cloud).

[0254] The feature recovery network decodes and processes the compressed features to recover the high-dimensional feature map. Through multiple layers of deconvolution and activation functions, this module can gradually recover and reconstruct the feature representation of the point cloud, providing high-quality features for subsequent point cloud reconstruction.

[0255] In an illustrative example, when the residual point cloud attribute features of three-level residual point clouds are written in the code stream, the feature recovery network corresponding to each level of residual point cloud and the bottom layer down-sampled point cloud is shown in FIG. 14.

[0256] The first feature recovery network 1401 corresponding to the first layer residual point cloud includes one layer of sparse deconvolution layer (TSConv 3 3TSConv 3 3 ×128 s2↑and SConv 3 3 ×128) and one activation layer (ReLU), and the activation layer is located between the two sparse deconvolution layers; the third feature recovery network 1403 corresponding to the third layer of residual point cloud includes three sparse deconvolution layers (TSConv 3 3 ×128 s2↑, TSConv 3 3 ×128 and TSConv 3 3 ×128) and two activation layers (ReLU), and the activation layers are arranged between adjacent sparse deconvolution layers; the fourth feature recovery network 1404 corresponding to the bottom layer of down-sampled point cloud includes five sparse deconvolution layers (TSConv 3 3 ×128 s2↑, TSConv 3 3 ×128, TSConv 3 3 ×128 s2↑, TSConv 3 3 ×128 and TSConv 3 3 ×128) and three activation layers (ReLU), and the activation layers are arranged between the first and second sparse deconvolution layers, and between the third, fourth and fifth sparse deconvolution layers. Among them, the sparse deconvolution layer in each feature recovery network uses a three-dimensional convolution kernel, and s2↑ represents two times up-sampling.

[0257] In some embodiments, the attribute reconstruction networks corresponding to different point clouds are the same, that is, the same attribute reconstruction network is connected after different feature recovery networks.

[0258] The input of the attribute reconstruction network is the residual point cloud attribute features output by the feature recovery network and the bottom layer of down-sampled point cloud attribute features. In order to restore the point cloud attribute features to the point cloud attribute information, the attribute reconstruction network needs to deconvolve the features.

[0259] And mirroring the feature extraction process at the encoding end, the deconvolution layer of the attribute reconstruction network uses sparse deconvolution and offset attention layer to improve the accuracy of local and global feature recovery.

[0260] In one possible design, the attribute reconstruction network includes a deconvolution layer, an activation layer and an offset attention layer.

[0261] In an illustrative example, as shown in FIG. 15, the attribute reconstruction network 1500 includes four sparse deconvolution layers (TSConv 3 3 ×128, TSConv 3 3 ×32, TSConv 3 3x16 and TSConv3 3 The feature extraction network comprises three sparse deconvolution layers, an activation layer, and an offset attention layer (Offset-AT), wherein the activation layer is located between the first two sparse deconvolution layers, and the offset attention layer is located between the last two sparse deconvolution layers.

[0262] In the above embodiment, configuring different depths of the feature recovery network for the residual point cloud at different levels and the bottom layer down-sampling point cloud helps to improve the feature recovery quality of the point cloud at different levels. Moreover, the feature reconstruction network adopts the sparse deconvolution and the offset attention mechanism, which can improve the reconstruction accuracy of the local feature and the global feature, and improve the point cloud reconstruction quality.

[0263] In a second mode, the code stream is decoded and feature recovery is performed to obtain attribute features of the residual point cloud; attribute reconstruction is performed based on the attribute features of the residual point cloud to obtain attributes of the reconstructed residual point cloud; and the code stream is decoded to determine attributes of the reconstructed bottom layer down-sampling point cloud.

[0264] The decoding reconstruction mode corresponds to the second encoding mode at the encoding end. In this mode, the residual point cloud is encoded into the code stream after feature extraction, and the bottom layer down-sampling point cloud is directly encoded into the code stream without feature extraction.

[0265] Correspondingly, the decoder decodes the code stream and performs feature recovery to obtain attribute features of the residual point cloud, and reconstructs attributes of the residual point cloud based on the attribute features of the residual point cloud; and decodes the code stream to determine attributes of the reconstructed bottom layer down-sampling point cloud.

[0266] Optionally, the decoder performs G-PCC attribute decoding on the code stream to obtain attributes of the bottom layer down-sampling point cloud. For example, the decoder performs entropy decoding, inverse quantization, and lifting inverse transform or RAHT inverse transform to obtain attribute values of the bottom layer down-sampling point cloud. The attribute decoding process of the specific point cloud is not limited in the embodiments of the present application.

[0267] In a possible implementation, the decoder decodes the code stream and performs feature recovery through the feature recovery network to obtain attribute features of the residual point cloud. The decoder performs attribute reconstruction through the attribute reconstruction network based on the attribute features of the residual point cloud to obtain attributes of the reconstructed residual point cloud.

[0268] Optionally, the feature recovery network and the attribute reconstruction network are mirror networks of the feature extraction network at the encoding end.

[0269] In some embodiments, in the case that the same feature extraction network is used at the encoding end for attribute feature extraction of the residual point cloud, the decoder also performs point cloud attribute feature recovery through the same feature recovery network.

[0270] In some other embodiments, when different feature extraction networks are used for different residual point clouds at the encoding end, the decoder also needs to use different feature recovery networks to recover the attribute features.

[0271] Optionally, the different feature recovery networks have different network depths.

[0272] In a possible implementation, the decoder decodes the code stream and recovers the attribute features of the residual point clouds through the feature recovery network corresponding to the residual point cloud.

[0273] Since a deeper network is used at the encoding end to extract features of the lower-level residual point clouds, a deeper network needs to be used at the decoding end to recover the features, i.e., the depth of the feature recovery network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud.

[0274] In an illustrative example, when the attribute features of three-level residual point clouds are written in the code stream, the depth of the feature recovery network corresponding to the third-level residual point cloud (the residual point cloud between the second-level down-sampled point cloud and the third-level down-sampled point cloud) is greater than the depth of the feature recovery network corresponding to the second-level residual point cloud (the residual point cloud between the first-level down-sampled point cloud and the second-level down-sampled point cloud), which is greater than the depth of the feature recovery network corresponding to the first-level residual point cloud (the residual point cloud between the original point cloud and the first-level down-sampled point cloud).

[0275] As to the structure of the feature recovery network, in a possible implementation, the feature recovery network includes a de-convolution layer and an activation layer. The number of layers of the de-convolution layer is positively correlated with the depth of the feature recovery network.

[0276] The number of layers of the de-convolution layer is positively correlated with the depth of the feature recovery network, i.e., the deeper the feature recovery network, the more the number of layers of the de-convolution layer. In the embodiments of the present application, the number of layers of the de-convolution layer in the feature recovery network corresponding to the bottom-level down-sampled point cloud is greater than the number of layers of the de-convolution layer in the feature recovery network corresponding to the residual point cloud; and the number of layers of the de-convolution layer in the feature recovery network corresponding to the i+1-level residual point cloud (the residual point cloud between the i-level down-sampled point cloud and the i+1-level down-sampled point cloud) is greater than the number of layers of the de-convolution layer in the feature recovery network corresponding to the i-level residual point cloud (the residual point cloud between the i-1-level down-sampled point cloud and the i-level down-sampled point cloud).

[0277] The feature recovery network decodes and processes the compressed features to recover the high-dimensional feature map. Through multiple de-convolution and activation functions, this module can gradually recover and reconstruct the feature representation of the point cloud, providing high-quality features for subsequent point cloud reconstruction.

[0278] In an illustrative example, when the residual point cloud attribute features of the 3-level residual point cloud are written in the code stream, the first feature recovery network corresponding to the first-level residual point cloud includes one sparse deconvolution layer; the second feature recovery network corresponding to the second-level residual point cloud includes two sparse deconvolution layers and one activation layer, and the activation layer is located between the two sparse deconvolution layers; and the third feature recovery network corresponding to the third-level residual point cloud includes three sparse deconvolution layers and two activation layers, and the activation layers are arranged between adjacent sparse deconvolution layers. In each feature recovery network, the sparse deconvolution layer uses a three-dimensional convolution kernel.

[0279] In some embodiments, the attribute reconstruction networks corresponding to different point clouds are the same, i.e., the same attribute reconstruction network is connected after different feature recovery networks.

[0280] The input of the attribute reconstruction network is the residual point cloud attribute features output by the feature recovery network. In order to restore the point cloud attribute features to the point cloud attribute information, the attribute reconstruction network needs to perform deconvolution on the features.

[0281] And mirroring the feature extraction process at the encoding end, the deconvolution layer of the attribute reconstruction network uses sparse deconvolution and offset attention layers to improve the accuracy of local and global feature recovery.

[0282] In a possible design, the attribute reconstruction network includes a deconvolution layer, an activation layer, and an offset attention layer. The structure of the attribute reconstruction network can refer to FIG. 15, which is not described herein.

[0283] For the above-described method one, when the hyper-prior parameters are written in the code stream at the encoding end, the decoder obtains the hyper-prior parameters from the code stream, decodes the code stream based on the hyper-prior parameters, and performs feature recovery through the feature recovery network to obtain the residual point cloud attribute features and the bottom-level down-sampled point cloud attribute features.

[0284] The hyper-prior parameters are determined by a hyper-prior model based on the bottom-level down-sampled point cloud attribute features. The hyper-prior parameters can include σ and μ.

[0285] In some embodiments, the decoder uses the hyper-prior parameters in the entropy decoding process of each level.

[0286] In the above-described embodiments, the hyper-prior model is introduced in the encoding process for entropy encoding, and the decoder obtains the hyper-prior parameters in the code stream and performs entropy decoding based only on the hyper-prior parameters, which helps to improve the decoding efficiency.

[0287] In combination with the above-described embodiments, in an illustrative example, as shown in FIG. 16, the point cloud attribute decoding process can include the following steps.

[0288] Step 1601, obtaining the hyper-prior parameter from the bitstream.

[0289] Step 1602, decoding the bitstream based on the hyper-prior parameter, performing feature recovery through the feature recovery network corresponding to the residual point cloud to obtain the attribute feature of the residual point cloud.

[0290] In some embodiments, the decoder entropy decodes the bitstream based on the hyper-prior parameter, and further performs feature recovery through the feature recovery network corresponding to the residual point cloud to obtain the attribute feature of the residual point cloud. Different levels of residual point clouds correspond to different depths of feature recovery networks.

[0291] As shown in FIG. 12, the decoder uses the first feature recovery network 1213 to recover the attribute feature of the first residual point cloud, uses the second feature recovery network 1214 to extract the attribute feature of the second residual point cloud, and uses the third feature recovery network 1215 to extract the attribute feature of the third residual point cloud.

[0292] Step 1603, decoding the bitstream based on the hyper-prior parameter, performing feature recovery through the feature recovery network corresponding to the bottom layer down-sampling point cloud to obtain the attribute feature of the bottom layer down-sampling point cloud.

[0293] In some embodiments, the decoder entropy decodes the bitstream based on the hyper-prior parameter, and further performs feature recovery through the feature recovery network corresponding to the bottom layer down-sampling point cloud to obtain the attribute feature of the bottom layer down-sampling point cloud. The depth of the feature recovery network corresponding to the bottom layer down-sampling point cloud is greater than the depth of the feature recovery network corresponding to the residual point cloud.

[0294] As shown in FIG. 12, the decoder uses the fourth feature recovery network 1216 to recover the attribute feature of the bottom layer down-sampling point cloud.

[0295] Step 1604, performing attribute reconstruction through the attribute reconstruction network based on the attribute feature of the residual point cloud and the attribute feature of the bottom layer down-sampling point cloud to obtain the reconstructed attribute of the residual point cloud and the reconstructed attribute of the bottom layer down-sampling point cloud.

[0296] As shown in FIG. 12, the decoder uses the attribute reconstruction network 1217 to perform attribute reconstruction on the attribute feature of the point cloud of different levels respectively to obtain the reconstructed first residual point cloud 1205, the second residual point cloud 1206, the third residual point cloud 1207, and the third layer down-sampling point cloud 1204.

[0297] Step 1605, performing layer-by-layer splicing on the reconstructed attribute of the bottom layer down-sampling point cloud and the reconstructed attribute of the residual point cloud to obtain the attribute of the reconstructed point cloud.

[0298] As shown in FIG. 12, the decoder splices the third layer down-sampled point cloud 1204 and the third residual point cloud 1207 to obtain a reconstructed second layer down-sampled point cloud 1203; splices the reconstructed second layer down-sampled point cloud 1203 and the second residual point cloud 1206 to obtain a reconstructed first layer down-sampled point cloud 1202; and splices the reconstructed first layer down-sampled point cloud 1202 and the first residual point cloud 1205 to obtain a reconstructed original point cloud 1201.

[0299] Regarding the training process of the network in each of the above embodiments, in a possible implementation, a comprehensive loss function is designed, which includes an entropy coding error, a coding-decoding error of each layer, and a hyper-prior error.

[0300] In some embodiments, the loss function is composed of two parts, a reconstruction error and an entropy coding error.

[0301] Reconstruction error

[0302] The reconstruction error is used to measure the difference between the point cloud attributes of the reconstructed point cloud and the point cloud attributes of the original point cloud. Optionally, a point-to-point mean squared error (MSE) can be used as a measurement method of the reconstruction error. Given the original point cloud color attribute C i and the reconstructed point cloud color attribute The reconstruction error is defined as:

[0303] where N is the number of points, C i and represent the original and reconstructed color attributes of the i-th point, respectively.

[0304] Entropy coding error

[0305] The entropy coding error is used to measure the compression rate of the encoded features. For the encoded features of each layer, the entropy coding cost is calculated. Optionally, when the hyper-prior parameter is used for coding and decoding, let the encoded features of the i-th layer be The entropy coding error is defined as:

[0306] where M i is the number of features of the i-th layer, is the probability distribution of the features based on the hyper-prior parameter .

[0307] Hyper-prior error

[0308] The super-prior error captures the difference between actual and predicted super-prior information, ensuring that the hypermodel effectively guides the entropy model. This loss helps refine the super-prior parameters μ and σ used in entropy encoding. The super-prior error can be defined as:

[0309] in, It is a priori Conditional probability model, It is a decomposition probability model, N L This represents the number of points in the bottom-level downsampling point cloud.

[0310] Total loss function

[0311] The total loss function is a weighted sum of the reconstruction errors of each layer, the entropy coding errors, and the prior errors. Optionally, when there are L layers of encoded features, the total loss function is defined as:

[0312] Dataset Construction

[0313] The network training uses three datasets: the COCO+Shapenet dataset (which projects COCO onto Shapenet), the Stanford3dDataset, and the RealWorldTexturedThings dataset. The network test dataset can use the standard test set published by MPEG.

[0314] Please refer to Figure 17, which shows a structural block diagram of a point cloud attribute encoding apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0315] The encoding unit 1701 is used to downsample the original point cloud multiple times to obtain multiple downsampled point clouds at different levels;

[0316] Encoding unit 1701 is used to extract attribute features from the residual point cloud between adjacent levels of point cloud to obtain residual point cloud attribute features.

[0317] The encoding unit 1701 is used to encode the attribute features of the residual point cloud and to encode the underlying downsampled point cloud.

[0318] Optionally, encoding unit 1701 is used for:

[0319] Determine the residual point cloud between adjacent point clouds;

[0320] The residual point cloud is subjected to attribute feature extraction to obtain the attribute features of the residual point cloud.

[0321] Optionally, encoding unit 1701 is used for:

[0322] determining a residual point cloud layer number;

[0323] determining the residual point cloud between point clouds of adjacent levels based on the residual point cloud layer number.

[0324] Optionally, the encoding unit 1701 is configured to:

[0325] in a case where the residual point cloud layer number is n layers and m layers of down-sampled point clouds are obtained by down-sampling, determining m-th to m-n+1-th residual point clouds, n being less than or equal to m;

[0326] wherein the first residual point cloud is the residual point cloud between the original point cloud and the first layer of down-sampled point clouds, the i-th residual point cloud is the residual point cloud between the i-th layer of down-sampled point clouds and the i-1-th layer of down-sampled point clouds, the first layer of down-sampled point clouds is obtained by down-sampling the original point cloud, the i-th layer of down-sampled point clouds is obtained by down-sampling the i-1-th layer of down-sampled point clouds, and i is an integer greater than or equal to 2.

[0327] Optionally, the encoding unit 1701 is configured to:

[0328] encoding the attribute features of the residual point clouds;

[0329] extracting attribute features of the bottom layer of down-sampled point clouds to obtain bottom layer down-sampled point cloud attribute features, and encoding the bottom layer down-sampled point cloud attribute features.

[0330] Optionally, the encoding unit 1701 is configured to:

[0331] extracting attribute features of the residual point clouds by using a feature extraction network corresponding to the residual point clouds to obtain the attribute features of the residual point clouds, wherein the depth of the feature extraction network corresponding to the residual point clouds is positively correlated with the level of the residual point clouds;

[0332] extracting attribute features of the bottom layer of down-sampled point clouds by using a feature extraction network corresponding to the bottom layer of down-sampled point clouds to obtain the bottom layer down-sampled point cloud attribute features, wherein the depth of the feature extraction network corresponding to the bottom layer of down-sampled point clouds is greater than the depth of the feature extraction network corresponding to the residual point clouds.

[0333] Optionally, the feature extraction network comprises a convolution layer, an activation layer, and a bias attention layer, and the number of layers of the convolution layer is positively correlated with the depth of the feature extraction network.

[0334] Optionally, the feature extraction network further comprises a feature fusion layer, the feature fusion layer is configured to determine normal information of the residual point cloud based on geometric information of the residual point cloud, and splice the normal information of the residual point cloud and shallow attribute features of the residual point cloud to obtain fused features, the shallow attribute features of the residual point cloud being output by the shallow convolution layer;

[0335] The feature fusion layer is further configured to determine normal information of the bottom layer down-sampling point cloud based on geometric information of the bottom layer down-sampling point cloud, and splice the normal information of the bottom layer down-sampling point cloud and shallow attribute features of the bottom layer down-sampling point cloud to obtain fused features, the shallow attribute features of the residual point cloud being output by the shallow convolution layer.

[0336] Optionally, the encoding unit 1701 is configured to:

[0337] determine a hyper-prior parameter based on the attribute features of the bottom layer down-sampling point cloud, the hyper-prior parameter being used for entropy encoding;

[0338] encode the hyper-prior parameter into a bitstream;

[0339] encode the attribute features of the residual point cloud based on the hyper-prior parameter;

[0340] encode the attribute features of the bottom layer down-sampling point cloud based on the hyper-prior parameter.

[0341] Optionally, the encoding unit 1701 is configured to determine hyper-prior information based on the attribute features of the bottom layer down-sampling point cloud;

[0342] determine an adaptive scaling factor based on the hyper-prior information;

[0343] perform adaptive quantization processing on the attribute features of the bottom layer down-sampling point cloud and the attribute features of the residual point cloud based on the adaptive scaling factor;

[0344] encode the attribute features of the residual point cloud after the adaptive quantization processing based on the hyper-prior parameter;

[0345] The encoding of the attribute features of the bottom layer down-sampling point cloud based on the hyper-prior parameter comprises:

[0346] encoding the attribute features of the bottom layer down-sampling point cloud after the adaptive quantization processing based on the hyper-prior parameter.

[0347] Optionally, the encoding unit 1701 is configured to:

[0348] perform octree partitioning on the residual point cloud and the bottom layer down-sampling point cloud to obtain hierarchical residual sub-point clouds and hierarchical down-sampling sub-point clouds;

[0349] perform attribute feature extraction on the hierarchical residual sub-point cloud to obtain residual point cloud attribute features;

[0350] perform attribute feature extraction on the hierarchical residual sub-point cloud to obtain residual point cloud attribute features;

[0351] Optionally, the residual point cloud and the bottom layer down-sampling point cloud are octree-divided according to the same octree depth.

[0352] Or,

[0353] The residual point cloud and the bottom layer down-sampling point cloud are octree-divided according to different octree depths, and different residual point clouds are octree-divided according to different octree depths, wherein the octree depth corresponding to the bottom layer down-sampling point cloud is less than the octree depth corresponding to the residual point cloud, and the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

[0354] Optionally, the encoding unit 1701 is configured to:

[0355] encode the residual point cloud attribute features;

[0356] perform attribute encoding on the bottom layer down-sampling point cloud.

[0357] Optionally, the encoding unit 1701 is configured to:

[0358] perform attribute feature extraction on the residual point cloud through a feature extraction network corresponding to the residual point cloud to obtain the residual point cloud attribute features, wherein the depth of the feature extraction network is positively correlated with the level of the residual point cloud.

[0359] Optionally, the feature extraction network comprises a convolution layer, an activation layer and a bias attention layer, and the number of layers of the convolution layer is positively correlated with the depth of the feature extraction network.

[0360] Optionally, the feature extraction network further comprises a feature fusion layer, the feature fusion layer is configured to determine normal information of the residual point cloud based on geometric information of the residual point cloud, and splice the normal information of the residual point cloud and shallow layer attribute features of the residual point cloud to obtain fused features, the shallow layer attribute features of the residual point cloud being output by the shallow layer convolution layer.

[0361] Optionally, the encoding unit 1701 is configured to:

[0362] perform octree division on the residual point cloud to obtain a hierarchical residual sub-point cloud;

[0363] Attribute feature extraction is performed on the hierarchical residual sub-point cloud to obtain residual point cloud attribute features.

[0364] Optionally, different residual point clouds are octree-divided according to the same octree depth.

[0365] Or,

[0366] Different residual point clouds are octree-divided according to different octree depths, wherein the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

[0367] Optionally, the input of the feature extraction network includes the geometric information and attribute information of the point cloud.

[0368] Optionally, the encoding unit 1701 is configured to:

[0369] The original point cloud is down-sampled multiple times by a low-frequency sampling module to obtain multiple down-sampled point clouds at different levels, and the low-frequency sampling module is configured to extract and filter low-frequency regions in the point cloud.

[0370] Or,

[0371] The original point cloud is sampled multiple times by a farthest point sampling module to obtain multiple down-sampled point clouds at different levels.

[0372] Optionally, the input of the low-frequency sampling module includes the geometric information and attribute information of the point cloud.

[0373] Optionally, the low-frequency sampling module includes a Hamming window processing submodule, an FFT submodule, a frequency domain sampling network, and an IFFT submodule.

[0374] The frequency domain sampling network is configured to extract and filter low-frequency frequency domain data, and the frequency domain sampling network includes an MLP layer, a global mean pooling layer, a self-attention layer, a convolution layer, and a max-pooling layer.

[0375] Please refer to FIG. 18, which shows a structural block diagram of a point cloud attribute decoding device according to an example embodiment of the present application. The device includes:

[0376] The decoding unit 1801 is configured to perform attribute decoding reconstruction based on the code stream to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampled point cloud.

[0377] The decoding unit 1801 is configured to perform layer-by-layer splicing on the attributes of the reconstructed bottom layer down-sampled point cloud and the attributes of the reconstructed residual point cloud to obtain the attributes of the reconstructed point cloud.

[0378] Optionally, the decoding unit 1801 is configured to:

[0379] decode the code stream and perform feature recovery to obtain residual point cloud attribute features and bottom layer down-sampling point cloud attribute features;

[0380] perform attribute reconstruction based on the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features to obtain attributes of the reconstructed residual point cloud and the reconstructed bottom layer down-sampling point cloud.

[0381] Optionally, the decoding unit 1801 is configured to:

[0382] decode the code stream and perform feature recovery through a feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features.

[0383] Optionally, the decoding unit 1801 is configured to:

[0384] obtain a hyper-prior parameter from the code stream, the hyper-prior parameter being used for entropy decoding;

[0385] decode the code stream based on the hyper-prior parameter and perform feature recovery through a feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features.

[0386] Optionally, the decoding unit 1801 is configured to:

[0387] perform feature recovery through a feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute features;

[0388] perform feature recovery through a feature recovery network corresponding to the bottom layer down-sampling point cloud to obtain the bottom layer down-sampling point cloud attribute features;

[0389] wherein a depth of the feature recovery network corresponding to the residual point cloud is positively correlated with a level of the residual point cloud, and a depth of the feature recovery network corresponding to the bottom layer down-sampling point cloud is greater than the depth of the feature recovery network corresponding to the residual point cloud.

[0390] Optionally, the feature recovery network comprises a deconvolution layer and an activation layer, and a number of layers of the deconvolution layer is positively correlated with the depth of the feature recovery network.

[0391] Optionally, the decoding unit 1801 is configured to:

[0392] perform attribute reconstruction through an attribute reconstruction network based on the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features to obtain attributes of the reconstructed residual point cloud and the reconstructed bottom layer down-sampling point cloud.

[0393] Optionally, the attribute reconstruction network comprises a deconvolution layer, an activation layer and an offset attention layer.

[0394] Optionally, the decoding unit 1801 is configured to:

[0395] decode the bitstream, and perform feature recovery based on the feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute feature.

[0396] perform attribute reconstruction based on the residual point cloud attribute feature to obtain the attribute of the reconstructed residual point cloud.

[0397] decode the bitstream to determine the attribute of the reconstructed bottom-level down-sampled point cloud.

[0398] Optionally, the decoding unit 1801 is configured to:

[0399] decode the bitstream, and perform feature recovery based on the feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute feature, wherein the depth of the feature recovery network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud.

[0400] Optionally, the feature recovery network comprises a deconvolution layer and an activation layer, and the number of layers of the deconvolution layer is positively correlated with the depth of the feature recovery network.

[0401] Optionally, the decoding unit 1801 is configured to:

[0402] perform attribute reconstruction based on the residual point cloud attribute feature to obtain the attribute of the reconstructed residual point cloud.

[0403] Optionally, the attribute reconstruction network comprises a deconvolution layer, an activation layer and an offset attention layer.

[0404] It should be noted that the process of attribute decoding implemented by the decoding unit described above can refer to the point cloud attribute decoding method embodiments described above, and the process of attribute encoding implemented by the encoding unit can refer to the point cloud attribute encoding method embodiments described above. Therefore, the present embodiment will not be described here.

[0405] Please refer to FIG. 19, which shows the structure block diagram of an encoder provided by an example embodiment of the present application. The encoder can comprise one or more of the following components: a processor 1901, a memory 1902. Each component is coupled together by a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0406] The memory 1902 is configured to store a computer program capable of running on the processor 1901; and the processor 1901 is configured to execute the following operations when running the computer program:

[0407] perform multiple down-sampling on the original point cloud to obtain multiple down-sampled point clouds of different levels;

[0408] Attribute feature extraction is performed on the residual point cloud between the point clouds of adjacent levels to obtain residual point cloud attribute features;

[0409] The residual point cloud attribute features are encoded, and the bottom layer down-sampling point cloud is encoded.

[0410] It can be understood that the memory 1902 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 1902 described in the present application is intended to include but not limited to these and any other suitable types of memory.

[0411] The processor 1901 can be an integrated circuit chip having a processing capability of signals. In the implementation process, the steps of the point cloud attribute encoding method described above can be completed by the integrated logic circuit of hardware in the processor 1901 or the instructions in the form of software. The processor 1901 described above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by hardware and software module combination in the code processor. The software module can be located in the random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, register or other mature storage medium in the art. The storage medium is located in the memory 1902, and the processor 1901 reads the information in the memory 1902, and combines the hardware to complete the steps of the method described above. It can be understood that the embodiments described in the present application can be realized by hardware, software, firmware, middleware, microcode or their combination. For hardware implementation, the processing unit can be realized in one or more ASIC, DSP, digital signal processing device (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), FPGA, general processor, controller, microcontroller, microprocessor, other electronic units for executing the functions described in the present application or their combination. For software implementation, the technology described in the present application can be realized by modules (such as processes, functions, etc.) for executing the functions described in the present application. The software code can be stored in the memory and executed by the processor. The memory can be implemented in the processor or outside the processor. Alternatively, as another embodiment, the processor 1901 is also configured to execute the point cloud attribute encoding method described in any one of the preceding embodiments when running the computer program.

[0412] Referring to FIG. 20, a structural block diagram of a decoder is shown according to an example embodiment of the present application. The decoder can include one or more of the following components: a processor 2001, a memory 2002. The various components are coupled together by a bus system, which can include a power bus, a control bus and a status bus. It will be appreciated that the bus system is configured to enable communications in each of the directions.

[0413] The memory 2002 is configured to store computer programs capable of running on the processor 2001; and the processor 2001 is configured to execute the following when running the computer programs:

[0414] Attribute decoding and reconstruction based on the bitstream to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed down-sampled point cloud of the base layer;

[0415] Layer-by-layer splicing of the attributes of the reconstructed down-sampled point cloud of the base layer and the attributes of the reconstructed residual point cloud to obtain the attributes of the reconstructed point cloud.

[0416] It can be understood that the memory 2002 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The description of the memory 2002 herein is intended to include, but not limited to, these and any other suitable types of memory.

[0417] The processor 2001 can be an integrated circuit chip having a processing capability of signals. In implementation, each step of the point cloud attribute decoding method described above can be completed by integrated logic circuits of hardware in the processor 2001 or instructions in the form of software. The processor 2001 described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 2002, and the processor 2001 reads the information in the memory 2002 and combines the hardware to complete the steps of the method described above. It can be understood that the embodiments described in the present application can be realized by hardware, software, firmware, middleware, microcode or their combination. For hardware implementation, the processing unit can be realized in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, general processors, controllers, microcontrollers, microprocessors, other electronic units for executing functions described in the present application or their combination. For software implementation, the technology described in the present application can be realized by modules (such as processes, functions, etc.) for executing functions described in the present application. The software code can be stored in the memory and executed by the processor. The memory can be implemented in the processor or outside the processor. Alternatively, as another embodiment, the processor 2001 is also configured to execute the point cloud attribute decoding method described in any one of the preceding embodiments when running the computer program.

[0418] The embodiment of the present application further provides a nonvolatile computer readable storage medium for storing a code stream, the code stream being generated by using the point cloud attribute encoding method of the encoder or being decoded by using the point cloud attribute decoding method of the decoder, wherein the point cloud attribute encoding method is the point cloud attribute encoding method described in the above embodiments, and the point cloud attribute decoding method is the point cloud attribute decoding method described in the above embodiments.

[0419] The embodiment of the present application further provides a computer program product, which comprises computer instructions stored in a computer readable storage medium, and a processor acquires the computer instructions from the computer readable storage medium, and executes the computer instructions to implement the point cloud attribute encoding method or the point cloud attribute decoding method described in the above embodiments.

[0420] Those skilled in the art can realize that, in one or more examples described above, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium accessible by a general or special purpose computer.

[0421] The above description is only optional embodiments of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of point cloud attribute decoding, the method comprising: The method comprises: Attribute decoding reconstruction based on the code stream to obtain attributes of the reconstructed residual point cloud and attributes of the reconstructed bottom layer down-sampling point cloud; Layer-by-layer splicing of the attributes of the reconstructed bottom layer down-sampling point cloud and the attributes of the reconstructed residual point cloud to obtain attributes of the reconstructed point cloud.

2. The method of claim 1, wherein, The attribute decoding reconstruction based on the code stream to obtain attributes of the reconstructed residual point cloud and attributes of the reconstructed bottom layer down-sampling point cloud comprises: Decoding the code stream and performing feature recovery to obtain residual point cloud attribute features and bottom layer down-sampling point cloud attribute features; Attribute reconstruction based on the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampling point cloud.

3. The method of claim 2, wherein, The decoding of the code stream and the feature recovery to obtain residual point cloud attribute features and bottom layer down-sampling point cloud attribute features comprises: Decoding the code stream and performing feature recovery through a feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features.

4. The method of claim 3, wherein, The decoding of the code stream and the feature recovery through the feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features comprises: Obtaining a hyper-prior parameter from the code stream, the hyper-prior parameter being used for entropy decoding; Decoding the code stream based on the hyper-prior parameter and performing feature recovery through the feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features.

5. The method of claim 3, wherein, The feature recovery through the feature recovery network to obtain the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features comprises: Performing feature recovery through a feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute features; Performing feature recovery through a feature recovery network corresponding to the bottom layer down-sampling point cloud to obtain the bottom layer down-sampling point cloud attribute features; The depth of the feature recovery network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud, and the depth of the feature recovery network corresponding to the bottom layer down-sampling point cloud is greater than the depth of the feature recovery network corresponding to the residual point cloud.

6. The method of claim 5, wherein, The feature recovery network comprises a deconvolution layer and an activation layer, and the number of layers of the deconvolution layer is positively correlated with the depth of the feature recovery network.

7. The method of claim 2, wherein, The attribute reconstruction based on the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampling point cloud comprises: Attribute reconstruction through an attribute reconstruction network based on the residual point cloud attribute features and the bottom layer down-sampling point cloud attribute features to obtain the attributes of the reconstructed residual point cloud and the attributes of the reconstructed bottom layer down-sampling point cloud.

8. The method of claim 7, wherein, The attribute reconstruction network comprises a deconvolution layer, an activation layer and a bias attention layer.

9. The method of claim 1, wherein, The attribute decoding reconstruction based on the code stream to obtain attributes of the reconstructed residual point cloud and attributes of the reconstructed bottom layer down-sampling point cloud comprises: Decoding the code stream and performing feature recovery to obtain residual point cloud attribute features; Attribute reconstruction based on the residual point cloud attribute features to obtain the attributes of the reconstructed residual point cloud; Decoding the code stream to determine the attribute of the reconstructed bottom layer down-sampled point cloud.

10. The method of claim 9, wherein, The decoding code stream and feature recovery are performed to obtain residual point cloud attribute features, including: Decoding the code stream and performing feature recovery through a feature recovery network corresponding to the residual point cloud to obtain the residual point cloud attribute features, the depth of the feature recovery network corresponding to the residual point cloud being positively correlated with the level of the residual point cloud.

11. The method of claim 10, wherein, The feature recovery network includes a deconvolution layer and an activation layer, and the number of layers of the deconvolution layer is positively correlated with the depth of the feature recovery network.

12. The method of claim 9, wherein, The attribute reconstruction based on the residual point cloud attribute features includes: The attribute reconstruction network includes a deconvolution layer, an activation layer, and an offset attention layer.

13. The method of claim 12, wherein, The method includes:

14. A method of encoding attributes of a point cloud, the method comprising: Multiple down-sampling of an original point cloud to obtain multiple down-sampled point clouds of different levels; Attribute feature extraction of residual point clouds between point clouds of adjacent levels to obtain residual point cloud attribute features; Encoding the residual point cloud attribute features and encoding the bottom layer down-sampled point cloud. The attribute feature extraction of the residual point clouds between the point clouds of adjacent levels to obtain the residual point cloud attribute features includes:

15. The method of claim 14, wherein, Determining the residual point cloud between the point clouds of adjacent levels; Attribute feature extraction of the residual point cloud to obtain the residual point cloud attribute features. The determination of the residual point cloud between the point clouds of adjacent levels includes:

16. The method of claim 15, wherein, Determining the number of residual point cloud layers; Based on the number of residual point cloud layers, the residual point cloud between the point clouds of adjacent levels is determined. Based on the number of residual point cloud layers, the residual point cloud between the point clouds of adjacent levels is determined.

17. The method of claim 16, wherein, In the case that the number of residual point cloud layers is n layers and m layers of down-sampled point clouds are obtained, the mth to the m-n+1th residual point clouds are determined, n being less than or equal to m; The first layer residual point cloud is the residual point cloud between the original point cloud and the first layer down-sampled point cloud, the ith layer residual point cloud is the residual point cloud between the ith layer down-sampled point cloud and the ith-1th layer down-sampled point cloud, the first layer down-sampled point cloud is obtained by down-sampling the original point cloud, the ith layer down-sampled point cloud is obtained by down-sampling the ith-1th layer down-sampled point cloud, and i is an integer greater than or equal to 2. The encoding of the residual point cloud attribute features and the encoding of the bottom layer down-sampled point cloud include:

18. The method of any one of claims 14 to 17, wherein, Encoding the residual point cloud attribute features; Attribute feature extraction of the bottom layer down-sampled point cloud to obtain bottom layer down-sampled point cloud attribute features; and encoding the bottom layer down-sampled point cloud attribute features. The attribute feature extraction of the residual point clouds between the point clouds of adjacent levels to obtain the residual point cloud attribute features includes:

19. The method of claim 18, wherein, Attribute feature extraction of the residual point cloud through a feature extraction network corresponding to the residual point cloud to obtain the residual point cloud attribute features, wherein the depth of the feature extraction network corresponding to the residual point cloud is positively correlated with the level of the residual point cloud. The attribute reconstruction network includes a deconvolution layer, an activation layer, and an offset attention layer. The attribute feature extraction is performed on the bottom layer down-sampling point cloud to obtain bottom layer down-sampling point cloud attribute features, and the attribute feature extraction comprises: The attribute feature extraction is performed on the bottom layer down-sampling point cloud to obtain bottom layer down-sampling point cloud attribute features, and the attribute feature extraction comprises:

20. The method of claim 19, wherein, The feature extraction network comprises a convolution layer, an activation layer and an offset attention layer, and the number of layers of the convolution layer is positively correlated with the depth of the feature extraction network.

21. The method of claim 20, wherein, The feature extraction network further comprises a feature fusion layer, which is configured to determine normal information of the residual point cloud based on the geometric information of the residual point cloud, and to splice the normal information of the residual point cloud and shallow layer attribute features of the residual point cloud to obtain fusion features, the shallow layer attribute features of the residual point cloud being output by the shallow layer convolution layer. The feature fusion layer is further configured to determine normal information of the bottom layer down-sampling point cloud based on the geometric information of the bottom layer down-sampling point cloud, and to splice the normal information of the bottom layer down-sampling point cloud and shallow layer attribute features of the bottom layer down-sampling point cloud to obtain fusion features, the shallow layer attribute features of the bottom layer down-sampling point cloud being output by the shallow layer convolution layer.

22. The method of claim 18, wherein, The method further comprises: determining a hyper-prior parameter based on the bottom layer down-sampling point cloud attribute features, the hyper-prior parameter being used for entropy coding; encoding the hyper-prior parameter into a bitstream; The encoding of the residual point cloud attribute features comprises: encoding the residual point cloud attribute features based on the hyper-prior parameter; The encoding of the bottom layer down-sampling point cloud attribute features comprises: encoding the bottom layer down-sampling point cloud attribute features based on the hyper-prior parameter.

23. The method of claim 22, wherein, The method further comprises: determining a hyper-prior information based on the bottom layer down-sampling point cloud attribute features; determining an adaptive scaling factor based on the hyper-prior information; performing adaptive quantization processing on the bottom layer down-sampling point cloud attribute features and the residual point cloud attribute features based on the adaptive scaling factor; The encoding of the residual point cloud attribute features based on the hyper-prior parameter comprises: encoding the residual point cloud attribute features after adaptive quantization processing based on the hyper-prior parameter; The encoding of the bottom layer down-sampling point cloud attribute features based on the hyper-prior parameter comprises: encoding the bottom layer down-sampling point cloud attribute features after adaptive quantization processing based on the hyper-prior parameter. The method further comprises:

24. The method of claim 18, wherein, performing octree division on the residual point cloud and the bottom layer down-sampling point cloud to obtain hierarchical residual sub-point clouds and hierarchical down-sampling sub-point clouds; The attribute feature extraction is performed on the bottom layer down-sampling point cloud to obtain bottom layer down-sampling point cloud attribute features, and the attribute feature extraction comprises: The attribute feature extraction is performed on the bottom layer down-sampling point cloud to obtain bottom layer down-sampling point cloud attribute features, and the attribute feature extraction comprises: ​ ​ Attribute feature extraction is performed on the hierarchical down-sampled point cloud to obtain an attribute feature of the bottom layer down-sampled point cloud.

25. The method of claim 24, wherein, the residual point cloud and the bottom layer down-sampled point cloud are octree partitioned according to the same octree depth; or, the residual point cloud and the bottom layer down-sampled point cloud are octree partitioned according to different octree depths, and different residual point clouds are octree partitioned according to different octree depths, wherein the octree depth corresponding to the bottom layer down-sampled point cloud is less than the octree depth corresponding to the residual point cloud, and the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

26. The method of any one of claims 14 to 17, wherein, The encoding of the attribute feature of the residual point cloud and the encoding of the bottom layer down-sampled point cloud comprise: encoding the attribute feature of the residual point cloud; encoding the attribute of the bottom layer down-sampled point cloud.

27. The method of claim 26, wherein, The attribute feature extraction of the residual point cloud between the point clouds of adjacent levels comprises: attribute feature extraction is performed on the residual point cloud by a feature extraction network corresponding to the residual point cloud to obtain the attribute feature of the residual point cloud, wherein the depth of the feature extraction network is positively correlated with the level of the residual point cloud.

28. The method of claim 27, wherein, The feature extraction network comprises a convolution layer, an activation layer, and a bias attention layer, and the number of layers of the convolution layer is positively correlated with the depth of the feature extraction network.

29. The method of claim 28, wherein, The feature extraction network further comprises a feature fusion layer, which is configured to determine normal information of the residual point cloud based on geometric information of the residual point cloud, and to splice the normal information of the residual point cloud and a shallow attribute feature of the residual point cloud to obtain a fusion feature, the shallow attribute feature of the residual point cloud being output by the shallow convolution layer.

30. The method of claim 26, wherein, The method further comprises: octree partitioning the residual point cloud to obtain hierarchical residual sub-point clouds; The attribute feature extraction of the residual point cloud between the point clouds of adjacent levels comprises: attribute feature extraction is performed on the hierarchical residual sub-point clouds to obtain the attribute feature of the residual point cloud.

31. The method of claim 30, wherein, different residual point clouds are octree partitioned according to the same octree depth; or, different residual point clouds are octree partitioned according to different octree depths, wherein the octree depth corresponding to the residual point cloud is negatively correlated with the level of the residual point cloud.

32. The method of claim 19 or 27, wherein, The input of the feature extraction network comprises geometric information and attribute information of the point cloud.

33. The method of any one of claims 14 to 32, wherein, The multiple down-sampling of the original point cloud to obtain multiple down-sampled point clouds of different levels comprises: multiple down-sampling of the original point cloud by a low-frequency sampling module to obtain multiple down-sampled point clouds of different levels, the low-frequency sampling module being configured to extract and filter low-frequency regions in the point cloud; or, multiple farthest point sampling of the original point cloud to obtain multiple down-sampled point clouds of different levels.

34. The method of claim 33, wherein, The input of the low-frequency sampling module comprises geometric information and attribute information of the point cloud.

35. The method of claim 33, wherein, The low-frequency sampling module comprises a Hamming window processing submodule, an FFT submodule, a frequency domain sampling network, and an IFFT submodule. The frequency domain sampling network is configured to extract and filter low-frequency frequency domain data, and comprises an MLP layer, a global mean pooling layer, a self-attention layer, a convolution layer, and a max-pooling layer.

36. An apparatus for point cloud attribute decoding, the apparatus comprising: The device comprises: The decoding unit is configured to perform attribute decoding reconstruction based on the bitstream to obtain attributes of a reconstructed residual point cloud and attributes of a reconstructed bottom layer down-sampled point cloud. The decoding unit is configured to perform layer-by-layer splicing on the attributes of the reconstructed bottom layer down-sampled point cloud and the attributes of the reconstructed residual point cloud to obtain attributes of a reconstructed point cloud.

37. An apparatus for point cloud attribute encoding, the apparatus comprising: The device comprises: The encoding unit is configured to perform multiple down-sampling on an original point cloud to obtain down-sampled point clouds of multiple different levels. The encoding unit is configured to perform attribute feature extraction on a residual point cloud between point clouds of adjacent levels to obtain residual point cloud attribute features. The encoding unit is configured to encode the residual point cloud attribute features and encode a bottom layer down-sampled point cloud.

38. A decoder, comprising: The decoder comprises a memory and a processor, the memory is configured to store a computer program running on the processor, and the processor is configured to execute the point cloud attribute decoding method according to any one of claims 1 to 13 when running the computer program.

39. An encoder comprising: The encoder comprises a memory and a processor, the memory is configured to store a computer program running on the processor, and the processor is configured to execute the point cloud attribute encoding method according to any one of claims 14 to 35 when running the computer program.

40. A non-transitory computer readable storage medium storing a bitstream, wherein, The bitstream is generated by using the point cloud encoding method of the encoder, or the bitstream is decoded by using the point cloud decoding method of the decoder, wherein the point cloud decoding method comprises the point cloud attribute decoding method according to any one of claims 1 to 13, or the point cloud encoding method comprises the point cloud attribute encoding method according to any one of claims 14 to 35.

41. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium, and a processor acquires the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to implement the point cloud attribute decoding method according to any one of claims 1 to 13, or the point cloud attribute encoding method according to any one of claims 14 to 35.

Images