Coding method, decoder, encoder and computer readable storage medium

Abstract

The embodiment of the application provides a coding and decoding method, a decoder, an encoder and a computer readable storage medium, which can improve coding and decoding efficiency and image reconstruction quality, and further improve coding and decoding performance. The method comprises the following steps: decoding the encoding information corresponding to the point cloud sequence in the code stream to determine the point cloud representation network parameter; the point cloud representation network parameter is determined by updating the network parameter of the initial network based on the time information and the space information corresponding to the point cloud sequence and the depth image sequence corresponding to the point cloud sequence by the encoder; based on the point cloud representation network parameter, the point cloud representation network is determined; at least one reconstructed depth image is determined according to the time information and the space information corresponding to the point cloud sequence by using the point cloud representation network; the time information represents the acquisition time corresponding to the point cloud sequence; the space information represents the space motion information of the acquisition equipment of the point cloud sequence; and at least one reconstructed point cloud corresponding to the point cloud sequence is determined based on the at least one reconstructed depth image.

Description

Coding and decoding method, decoder, encoder and computer-readable storage medium Technical Field

[0001] The present application relates to point cloud compression coding and decoding technology, and in particular to a coding and decoding method, a decoder, an encoder and a computer-readable storage medium. Background Art

[0002] A point cloud is a type of three-dimensional data, referring to a collection of vectors in a three-dimensional coordinate system. These vectors are typically represented as (x, y, z) coordinates and can also represent information such as color, material, and reflection intensity. They are generally used to represent three-dimensional objects or scenes. Laser Detection and Ranging (LiDAR) is a remote sensing technology that uses laser pulses to generate 3D point clouds, providing precise structural information about a scene. LiDAR point cloud sequences are generated by continuous scanning at the sensor's sampling frequency. With the rapid development of emerging technologies such as augmented reality, virtual reality, autonomous driving, and robotics, point cloud data has become one of the primary data formats due to its concise representation of three-dimensional space. Autonomous driving, robotic navigation, surveying and mapping modeling, and other fields all require LiDAR point cloud representation. However, point cloud data is massive, and directly storing point cloud data consumes a large amount of memory and is inconvenient for transmission.

[0003] Currently, the two representative point cloud compression algorithms are two technical solutions developed by the Moving Picture Experts Group (MPEG): Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). G-PCC compression is primarily achieved through an octree model and / or a triangular surface model. While this method supports LiDAR point cloud compression, it only supports single-frame point cloud compression, and its decoding time is slow, resulting in poor quality of the reconstructed point cloud. This reduces the efficiency and quality of point cloud encoding and decoding, and thus reduces point cloud encoding and decoding performance.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a computer-readable storage medium, which can improve the efficiency of point cloud coding and decoding, thereby improving coding and decoding performance.

[0006] The technical solution of this application is achieved as follows:

[0007] This embodiment of the present application provides a decoding method, including:

[0008] Decoding the encoded information corresponding to the point cloud sequence in the bitstream to determine the point cloud representation network parameters; the point cloud representation network parameters are determined by the encoder updating the network parameters of the initial network based on the temporal information and spatial information corresponding to the point cloud sequence and the depth map sequence corresponding to the point cloud sequence;

[0009] Determining a point cloud representation network based on the point cloud representation network parameters;

[0010] Determining at least one reconstructed depth image using the point cloud representation network based on time information and spatial information corresponding to the point cloud sequence, wherein the time information represents the acquisition time corresponding to the point cloud sequence; and the spatial information represents spatial motion information of an acquisition device for the point cloud sequence;

[0011] At least one reconstructed point cloud corresponding to the point cloud sequence is determined based on the at least one reconstructed depth image.

[0012] This embodiment of the present application provides an encoding method, including:

[0013] Performing coordinate transformation mapping on the point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence;

[0014] Performing spatiotemporal position encoding on the time information and spatial information corresponding to the point cloud sequence to determine a spatiotemporal encoding vector sequence corresponding to the depth map sequence; the time information represents the acquisition time corresponding to the point cloud sequence; the spatial information represents the spatial motion information of the acquisition device of the point cloud sequence;

[0015] Using the depth map sequence and the spatiotemporal coding vector sequence, updating the network parameters of the initial network to determine the point cloud representation network parameters;

[0016] The point cloud representation network parameters are encoded to determine encoding information corresponding to the point cloud sequence.

[0017] An embodiment of the present application provides a decoder, including:

[0018] The decoding part is configured to decode the encoded information corresponding to the point cloud sequence in the code stream and determine the point cloud representation network parameters; the point cloud representation network parameters are determined by the encoder updating the network parameters of the initial network based on the temporal information and spatial information corresponding to the point cloud sequence and the depth map sequence corresponding to the point cloud sequence;

[0019] a network parameter determination part, configured to determine a point cloud representation network based on the point cloud representation network parameters;

[0020] An image reconstruction part is configured to use the point cloud representation network to determine at least one reconstructed depth image according to the temporal information and spatial information corresponding to the point cloud sequence;

[0021] The point cloud reconstruction part is configured to determine at least one reconstructed point cloud corresponding to the point cloud sequence based on the at least one reconstructed depth image.

[0022] An embodiment of the present application provides an encoder, including:

[0023] A mapping part is configured to perform coordinate transformation mapping on the point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence;

[0024] A spatiotemporal coding portion is configured to perform spatiotemporal position coding on the time information and spatial information corresponding to the point cloud sequence, and determine a spatiotemporal coding vector sequence corresponding to the depth map sequence; the time information represents the acquisition time corresponding to the point cloud sequence; and the spatial information represents the spatial motion information of the acquisition device of the point cloud sequence;

[0025] A parameter updating part is configured to update the network parameters of the initial network using the depth map sequence and the spatiotemporal coding vector sequence to determine the point cloud representation network parameters;

[0026] The encoding part is configured to encode the point cloud representation network parameters and determine the encoding information corresponding to the point cloud sequence.

[0027] An embodiment of the present application provides a code stream, which is generated by bit encoding based on coding information; wherein the coding information at least includes: coding information corresponding to a point cloud sequence; the coding information corresponding to the point cloud sequence is obtained by encoding point cloud representation network parameters; the point cloud representation network parameters are determined by updating the network parameters of the initial network using a depth map sequence corresponding to the point cloud sequence, and time information and spatial information corresponding to the point cloud sequence.

[0028] An embodiment of the present application provides a decoder, including:

[0029] a first memory configured to store executable instructions;

[0030] The first processor is configured to implement any one of the above decoding methods when executing the executable instructions stored in the first memory.

[0031] An embodiment of the present application provides an encoder, including:

[0032] a second memory configured to store executable instructions;

[0033] The second processor is configured to implement any one of the encoding methods described above when executing the executable instructions stored in the second memory.

[0034] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a first processor to execute the above-mentioned decoding method, or for causing a second processor to execute the above-mentioned encoding method.

[0035] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a first processor, the decoding method provided by the embodiment of the present application is implemented; or, when the computer program or instructions are executed by a second processor, the encoding method provided by the embodiment of the present application is implemented.

[0036] The embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a computer-readable storage medium. The decoder determines a point cloud representation network based on the point cloud representation network parameters transmitted by the encoder, uses the point cloud representation network and the temporal and spatial information corresponding to at least one point cloud to reconstruct at least one depth image, and then recovers at least one reconstructed point cloud based on the at least one depth image. This implements decoding and reconstruction of a depth image represented by implicit neural network parameters using a neural network based on spatiotemporal position coding. This method can more fully leverage the advantages of the neural network, significantly reduce computational complexity, improve decoding efficiency, and enhance the quality of reconstruction after decoding. Furthermore, obtaining the high-frequency components of the point cloud representation network based on the temporal and spatial information corresponding to at least one point cloud can enhance the representation effect of the neural network, further improve reconstruction quality, and thereby enhance decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of an optional flow chart of an encoding method provided in an embodiment of the present application;

[0038] FIG2 is a schematic diagram of a process of mapping a point cloud into a depth map according to an embodiment of the present application;

[0039] FIG3 is a schematic diagram of an optional flow chart of an encoding method provided in an embodiment of the present application;

[0040] FIG4 is a schematic diagram of an optional process for training an initial network and encoding network parameters according to an embodiment of the present application;

[0041] FIG5 is a schematic diagram of an optional flow chart of a decoding method provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of an optional flow chart of a decoding method provided in an embodiment of the present application;

[0043] FIG7 is a schematic diagram of an optional process for decoding using a point cloud representation network according to an embodiment of the present application;

[0044] FIG8 is a schematic diagram of an optional process for encoding and decoding a noisy point cloud to achieve a denoising effect, provided in an embodiment of the present application;

[0045] FIG9 is a schematic diagram of an optional process for encoding, decoding, reconstructing, and segmenting a point cloud according to an embodiment of the present application;

[0046] FIG10A is a first diagram comparing the semantic segmentation effects of the present application and the current G-PCC provided in an embodiment of the present application;

[0047] FIG10B is a second comparison diagram of the semantic segmentation effects of the present application and the current G-PCC provided in an embodiment of the present application;

[0048] FIG11 is a schematic diagram of an optional structure of a decoder provided in an embodiment of the present application;

[0049] FIG12 is a schematic diagram of an optional structure of an encoder provided in an embodiment of the present application;

[0050] FIG13 is a schematic diagram of an optional structure of a decoder provided in an embodiment of the present application;

[0051] FIG14 is a schematic diagram of an optional structure of an encoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] Point cloud compression algorithms include: Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC). Among them, point cloud compression in G-PCC is mainly implemented based on the octree model and / or triangular surface model. V-PCC is mainly based on 3D to 2D projection and video compression. Although the current G-PCC compression method supports lidar point cloud compression, it only supports single-frame point cloud compression, and its decoding time is slow. At the same time, although implicit neural networks have great potential in both 3D and 2D signals, because point clouds have many points and are irregularly distributed, directly feeding the point cloud into the implicit neural network requires a long rendering time and the effect is very limited. Therefore, neural networks have not yet been applied to point cloud compression.

[0057] The embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a computer-readable storage medium. By using depth images to compactly represent radar point clouds, implicit neural networks can better function. Compared with directly using implicit neural networks to represent three-dimensional point clouds, the complexity of operations can be greatly reduced and the quality of compressed reconstruction can be improved. In addition, the redundancy in the time domain of the lidar point cloud sequence can be effectively extracted, which is more conducive to the modeling and recovery of image signals, thereby improving coding and decoding efficiency and point cloud reconstruction quality, thereby improving coding and decoding performance. The following describes the coding method applied to the encoder provided in the embodiments of the present application.

[0058] Refer to Figure 1, which is an optional flow chart of the encoding method provided in an embodiment of the present application, and will be explained in conjunction with the steps shown in Figure 1.

[0059] S101: Perform coordinate transformation mapping on a point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence.

[0060] In an embodiment of the present application, the point cloud sequence includes at least one (frame) point cloud; the encoder performs coordinate transformation mapping on each point cloud in the at least one point cloud, maps each point cloud into a depth map, and thereby determines a depth map sequence corresponding to the point cloud sequence.

[0061] In some embodiments, the coordinate transformation mapping process can be shown in Figure 2. The three-dimensional point cloud can be regarded as a representation of the sensor data provided by the lidar scanner. Each point in the point cloud corresponds to the measurement value of a single lidar beam, and each point can be described by a three-dimensional space coordinate (x, y, z) and its attribute information. The depth map can be regarded as another representation of the lidar scan, which saves the points in the three-dimensional space as points in a 360-degree two-dimensional image of the scanned environment. Among them, the row dimension represents the elevation angle of the laser beam, and the column dimension represents the azimuth angle. With each incremental rotation around the z-axis, the lidar sensor returns many distance and intensity measurements, which are then stored in the corresponding cells of the depth image. As shown in Figure 2, a point p in space is mapped to a depth image cell, which is represented by the corresponding azimuth angle θp and inclination angle θp. Indicates that the distance rangep of point p is stored in the corresponding cell. In this way, by performing coordinate mapping transformation on each point in the three-dimensional space and mapping it to a point in the depth map using the depth map projection method, the depth map corresponding to the point cloud is obtained.

[0062] The embodiment of the present application converts a point cloud sequence into a two-dimensional depth image sequence, making it more efficient and compact in spatial arrangement. At the same time, the two-dimensional expression can be more conveniently represented using an implicit neural network. It can be understood that converting the point cloud into a depth map and then performing an implicit neural representation of the depth map allows the implicit neural network to give full play to its advantages. Compared with directly using implicit neural representation of three-dimensional point clouds, it can greatly reduce the complexity of the calculation and achieve better reconstruction quality.

[0063] S102: Perform spatiotemporal position encoding on the time information and spatial information corresponding to the point cloud sequence to determine a spatiotemporal encoding vector sequence corresponding to the depth map sequence.

[0064] In an embodiment of the present application, the time information represents the acquisition time corresponding to the point cloud sequence; the spatial information represents the spatial motion information of the acquisition device of the point cloud sequence. Each point cloud in the point cloud sequence is acquired by an acquisition device, such as a device equipped with a lidar sensor, moving in a certain space. That is to say, each point cloud in the point cloud sequence corresponds to the acquisition time when the acquisition device acquires the point cloud, and the spatial motion information of the acquisition device. Here, the acquisition time is used as the time information corresponding to each point cloud, and the spatial motion information of the acquisition device, such as the spatial position and posture of the acquisition device when acquiring each point cloud, is used as the spatial information corresponding to each point cloud. The time information and spatial information corresponding to the point cloud sequence include: the time information corresponding to each point cloud and the spatial information corresponding to each point cloud.

[0065] In an embodiment of the present application, the encoder performs position encoding on the time information and spatial information corresponding to each point cloud in the point cloud sequence, and encodes the time information and spatial information corresponding to each point cloud into a high-dimensional vector as the space-time coding vector corresponding to each point cloud, that is, the space-time coding vector corresponding to each depth map, thereby determining the space-time coding vector sequence corresponding to the depth map sequence.

[0066] For example, the encoder can perform spatiotemporal position encoding on the acquisition time stamp of each depth map, as well as the spatial translation and rotational positions of the sensor on the acquisition device, to obtain a corresponding high-dimensional position encoding vector, which serves as the spatiotemporal encoding vector for each depth map. This improves the high-frequency classification of the input neural network, helps the neural network model the spatiotemporal relationship between lidars, and improves the reconstruction quality of the point cloud.

[0067] S103: Using the depth map sequence and the spatiotemporal coding vector sequence, update the network parameters of the initial network to determine the point cloud representation network parameters.

[0068] In an embodiment of the present application, the encoder uses a spatiotemporal coding vector sequence and a depth map sequence to train the initial network, and iteratively updates the network parameters of the initial network during the training process until the training goal is reached, and determines the most recently updated network parameters as the point cloud representation network parameters.

[0069] In some embodiments, the encoder reconstructs each depth map in the depth map sequence according to the spatiotemporal coding vector corresponding to each depth map in the spatiotemporal coding vector sequence through the initial network, and determines the reconstructed depth image corresponding to each depth map; based on the reconstructed depth image and each depth map, the network parameters of the initial network are iteratively updated until the preset update conditions are met, and then the point cloud representation network parameters are determined based on the currently updated network parameters. That is, in each network training, the initial network will perform image restoration and reconstruction on each depth map according to the spatiotemporal coding vector corresponding to each depth map in the depth map sequence, and determine the corresponding reconstructed depth image; the encoder updates the network parameters of the current training based on the difference between the reconstructed depth image and each depth map; if the preset update conditions are not met, the next training is performed based on the updated network parameters; until the preset update conditions are met, that is, when the training target is reached, the network parameters updated in the current training, that is, the currently updated network parameters, are used as the point cloud representation network parameters.

[0070] In some embodiments, the network parameters may include weights of the initial network; the preset update conditions may include: the difference between the reconstructed depth image and the corresponding depth map satisfies an overfitting condition, or the number of updates reaches a prediction threshold, etc. The specific selection is based on actual conditions and is not limited in the embodiments of this application.

[0071] In some embodiments, the initial network includes a multi-layer perceptron (MLP) and multiple cascaded convolutional neural network (CNN) layers. The MLP is used to represent the initial depth map signal, and the multiple cascaded convolutional layers are used to perform convolution recovery based on the initial depth map signal to determine a reconstructed depth image.

[0072] It can be understood that the embodiment of the present application uses the depth map sequence and the spatiotemporal coding vector sequence to update the network parameters of the initial network, which can realize the representation of the depth image through the implicit neural network and improve the representation efficiency of the LiDAR point cloud sequence.

[0073] S104: Encode the point cloud representation network parameters to determine the encoding information corresponding to the point cloud sequence.

[0074] In an embodiment of the present application, the encoder performs lossless encoding on the point cloud representation network parameters, determines the encoding information corresponding to the point cloud sequence, and generates a corresponding code stream to be sent to the decoder.

[0075] Exemplarily, the encoder may perform Huffman encoding on the point cloud representation network parameters to obtain encoding information corresponding to the point cloud sequence.

[0076] In some embodiments, the encoder can generate a bitstream based on the network parameter encoding information corresponding to the point cloud sequence, and send the temporal information corresponding to the point cloud sequence, the spatial information corresponding to the point cloud sequence, and the bitstream to the decoder. In other words, the bitstream includes the network parameter encoding information corresponding to the point cloud sequence to generate the bitstream, and the temporal information and spatial information corresponding to the point cloud sequence do not occupy additional bitstream.

[0077] It is understandable that the encoder converts the point cloud into a depth map, making it more efficient and compact in spatial arrangement; through spatiotemporal position encoding, the depth image is represented by an implicit neural network, further improving the representation efficiency of the point cloud sequence; thereby reducing computational complexity and improving encoding efficiency. In addition, the encoder of the embodiment of the present application, based on the position encoding of time information, also combines the spatial posture information of the sensor (such as translation and rotation) to splice the encoding vectors, and together as the input of the implicit neural network, can obtain a better representation effect, thereby improving the encoding quality and further improving the encoding performance.

[0078] In some embodiments, based on FIG. 1 , as shown in FIG. 3 , S102 may be implemented by executing the processes of S1021 to S1023 as follows:

[0079] S1021. For each point cloud in the point cloud sequence, position encode the time information corresponding to each point cloud, and determine a time encoding vector corresponding to each depth map.

[0080] S1022: Perform position encoding on the spatial information corresponding to each point cloud, and determine the spatial encoding vector corresponding to each depth map.

[0081] In the embodiment of the present application, the encoder performs position encoding on the time information and spatial information corresponding to each depth map, respectively, and determines a time coding vector and a spatial coding vector corresponding to each depth map.

[0082] In some embodiments, the spatial information includes translation information and rotation information. Here, the translation information represents the spatial translation position of an acquisition device, such as a sensor, and the rotation information represents the spatial rotation position of the acquisition device. The encoder can encode the translation information to determine a translation encoding vector; encode the rotation information to determine a rotation encoding vector; and determine the translation encoding vector and the rotation encoding vector as a spatial encoding vector.

[0083] S1023. Merge the temporal coding vector and the spatial coding vector to determine the spatiotemporal coding vector corresponding to each depth map, thereby determining a spatiotemporal coding vector sequence.

[0084] In this embodiment of the present application, the encoder combines the temporal and spatial coding vectors determined by independent position coding. For example, the event coding vector, translation coding vector, and rotation coding vector are vector-concatenated to obtain the spatiotemporal coding vector corresponding to each depth map. The encoder uses the same process to perform spatiotemporal position coding on the temporal and spatial information corresponding to each point cloud in the point cloud sequence to determine a spatiotemporal coding vector sequence.

[0085] Exemplarily, the encoder may include a spatiotemporal position encoding module, which may include a temporal position encoding module, a spatial translation position encoding module, and a spatial rotation position encoding module. The encoder normalizes the timestamp (time information), the sensor spatial translation position (translation information), and the sensor spatial rotation position (rotation information), respectively, and inputs each into the corresponding position encoding module to convert them into corresponding high-dimensional vectors. The three high-dimensional vectors are then concatenated to form a spatiotemporal encoding vector.

[0086] It can be understood that the embodiments of the present application fully consider that in actual applications, the sampling process of the lidar point cloud sequence is usually obtained by an acquisition device equipped with a lidar sensor moving in a certain space, and the time information is combined with the spatial posture information of the acquisition device, such as the encoding of translation and rotation, as the input of the implicit neural network, thereby improving the neural network's representation effect on the image and improving the performance in multiple downstream visual tasks.

[0087] In some embodiments, the encoder may further, upon meeting preset update conditions, prune the initial network and / or quantize network parameters based on the currently updated network parameters to determine the point cloud representation network parameters. Here, the encoder may reduce the network parameters (weights) corresponding to network pathways that have a smaller impact on the reconstructed depth image to zero within the currently updated network parameters, thereby pruning the initial network to obtain pruned network parameters; and / or the encoder may quantize the currently updated network parameters or the pruned network parameters, and determine the obtained quantized network parameters as the point cloud representation network parameters.

[0088] Exemplarily, as shown in FIG4 , the encoder uses a depth map sequence and a spatiotemporal coding vector sequence to perform network overfitting training on the initial network, iteratively updates the network parameters of the initial network during the training process, and performs network pruning (pruning) on ​​the initial network when the preset update conditions are met. The pruned network parameters are quantized, and the quantized network parameters are weight-encoded to determine the point cloud representation network parameters.

[0089] It is understandable that pruning and quantizing network parameters can further compress the network parameters that need to be encoded and improve encoding efficiency.

[0090] The following describes a decoding method applied to a decoder provided in an embodiment of the present application.

[0091] Refer to Figure 5, which is an optional flow chart of the decoding method provided in an embodiment of the present application, and will be explained in conjunction with the steps shown in Figure 5.

[0092] S201: Decode the encoding information corresponding to the point cloud sequence in the code stream to determine the point cloud representation network parameters.

[0093] In this embodiment of the present application, the decoder parses the encoded information corresponding to the point cloud sequence from the bitstream, decodes the encoded information corresponding to the point cloud sequence, and determines the point cloud representation network parameters. Here, the point cloud representation network parameters are determined by the encoder by updating the network parameters of the initial network based on the temporal and spatial information corresponding to the point cloud sequence and the depth map sequence corresponding to the point cloud sequence. The process of determining the point cloud representation network parameters is similar to the encoding process described on the encoder side and will not be repeated here.

[0094] In some embodiments, the decoder may perform lossless Huffman decoding on the encoded information corresponding to the point cloud sequence to determine the point cloud representation network parameters.

[0095] In some embodiments, the decoder receives the temporal information corresponding to the point cloud sequence, the spatial information corresponding to the point cloud sequence, and the bitstream sent by the encoder. Thus, the decoder parses and decodes the bitstream to determine the point cloud representation network parameters, and receives the temporal information corresponding to the point cloud sequence and the spatial information corresponding to the point cloud sequence.

[0096] S202: Determine a point cloud representation network based on point cloud representation network parameters.

[0097] In the embodiment of the present application, the decoder includes an initial network whose network architecture corresponds to the network architecture of the initial network in the encoder. The decoder determines the initial network and updates the network parameters of the initial network using the point cloud representation network parameters obtained by decoding to determine the point cloud representation network.

[0098] In some embodiments, the initial network includes: a multi-layer perceptron and a multi-layer cascaded convolutional layer, the multi-layer perceptron is used to express the initial depth map signal; the multi-layer cascaded convolutional layer is used to perform convolution recovery based on the initial depth map signal to determine the reconstructed depth image.

[0099] S203: Utilize the point cloud representation network to determine at least one reconstructed depth image according to the temporal information and spatial information corresponding to the point cloud sequence.

[0100] In this embodiment of the present application, the temporal information represents the acquisition time of the point cloud sequence, and the spatial information represents the spatial motion information of the acquisition device of the point cloud sequence. The temporal information and spatial information corresponding to the point cloud sequence include: the temporal information corresponding to each point cloud in the point cloud sequence; the spatial information corresponding to the point cloud sequence includes: the spatial information corresponding to each point cloud.

[0101] In an embodiment of the present application, the decoder utilizes a point cloud representation network to determine at least one reconstructed depth image based on the temporal and spatial information corresponding to at least one point cloud in the point cloud sequence. In other words, the decoder can determine the reconstructed depth image corresponding to each point cloud in the entire point cloud sequence based on the temporal and spatial information corresponding to all point clouds in the point cloud sequence, as the at least one reconstructed depth image. Alternatively, the decoder can determine the reconstructed depth image corresponding to each point cloud in a portion of the point cloud sequence based on the temporal and spatial information corresponding to the portion of the point cloud sequence, as the at least one reconstructed depth image, depending on actual decoding needs.

[0102] In this embodiment of the present application, the point cloud representation network is determined based on the point cloud representation network parameters transmitted by the encoder. These point cloud representation network parameters represent the depth map in the form of neural network parameters. Therefore, the decoder can utilize the point cloud representation network to reconstruct at least one reconstructed depth image corresponding to the at least one spatiotemporal code vector based on the at least one spatiotemporal code vector.

[0103] S204: Determine at least one reconstructed point cloud corresponding to the point cloud sequence based on the at least one reconstructed depth image.

[0104] In an embodiment of the present application, the decoder may recover the reconstructed point cloud corresponding to each reconstructed depth image in at least one reconstructed depth image, thereby determining at least one reconstructed point cloud.

[0105] In some embodiments, coordinate inverse mapping is performed on the at least one reconstructed depth image to determine at least one reconstructed point cloud.

[0106] It can be understood that the decoder determines the point cloud representation network based on the point cloud representation network parameters transmitted by the encoder, uses the point cloud representation network and the temporal and spatial information corresponding to at least one point cloud to reconstruct at least one depth image, and then restores at least one reconstructed point cloud based on the at least one depth image. This implements the decoding and reconstruction of the depth image represented by the implicit neural network parameters using a neural network based on spatiotemporal position coding. Therefore, it can more fully utilize the advantages of the neural network, greatly reduce the complexity of the calculation, improve the decoding efficiency, and improve the quality of the decoded reconstruction. In addition, the high-frequency components of the point cloud representation network are obtained based on the temporal and spatial information corresponding to at least one point cloud, which can improve the representation effect of the neural network, further improve the reconstruction quality, and thus improve the decoding performance.

[0107] In some embodiments, based on FIG. 5 , as shown in FIG. 6 , S203 may be implemented by executing the process of S2031 - S2032 as follows:

[0108] S2031. The decoder performs spatiotemporal position encoding on at least one time information and at least one spatial information corresponding to at least one point cloud in the point cloud sequence, and determines at least one spatiotemporal encoding vector.

[0109] In an embodiment of the present application, the decoder includes a space-time coding module, which can perform space-time position coding on at least one time information and at least one spatial information corresponding to at least one point cloud to determine at least one space-time coding vector.

[0110] In some embodiments, for each point cloud in at least one point cloud, the decoder performs position encoding on the time information corresponding to each point cloud to determine the time encoding vector corresponding to each point cloud; performs position encoding on the spatial information corresponding to each point cloud to determine the space encoding vector corresponding to each point cloud; merges the time encoding vector and the space encoding vector to determine the space-time encoding vector corresponding to each point cloud, thereby determining at least one space-time encoding vector.

[0111] In some embodiments, the spatial information includes translation information and rotation information; the spatial information corresponding to each point cloud is position-encoded, and the decoder encodes the translation information to determine a translation code vector; encodes the rotation information to determine a rotation code vector; and the translation code vector and the rotation code vector are determined as a spatial code vector. The decoder concatenates the temporal code vector, the translation code vector, and the rotation code vector corresponding to each point cloud in at least one point cloud to determine at least one spatiotemporal code vector.

[0112] S2032: Utilize the point cloud representation network to perform restoration and reconstruction according to at least one spatiotemporal coding vector to determine at least one reconstructed depth image.

[0113] Exemplarily, as shown in Figure 7, the temporal and spatial information corresponding to the depth map includes: time information t, translation information (x, y, z), and rotation information (α, β, γ). The encoder inputs the time information t, translation information (x, y, z), and rotation information (α, β, γ) into the spatiotemporal coding module for position encoding, and determines the spatiotemporal coding vector corresponding to the depth map. The encoder inputs the spatiotemporal coding vector into the initial network. The multi-layer perceptron of the initial network expresses the initial depth map signal based on the spatiotemporal coding vector, such as an image signal with a lower resolution than the original depth map. The initial network then uses the multi-layer cascaded convolutional layers to perform step-by-step convolution recovery based on the initial depth map signal to determine the reconstructed depth map corresponding to the depth map. The decoder performs coordinate inverse mapping based on the reconstructed depth map to determine the reconstructed point cloud.

[0114] It can be understood that the method of the embodiment of the present application can support geometric lossy compression of point clouds and has excellent performance in data compression. As shown in Table 1, the BD Rate is used to quantitatively evaluate the lossy compression. In terms of D1 and D2 PSNR on sequences #00, #02 and #08 of the SemanticKITTI dataset, in pure geometric lidar point cloud compression, the embodiment of the present application provides an average performance gain of 60.39% and 47.78% relative to G-PCC. This data shows that the compression efficiency of the embodiment of the present application significantly exceeds the current point cloud codec G-PCC, greatly improving the encoding and decoding performance. At the same time, the decoding speed of the embodiment of our application is higher than that of G-PCC, and only the corresponding time information such as timestamp needs to be input to decode and reconstruct the point cloud on the corresponding time information, supporting real-time decoding processing.

[0115] Table 1

[0116] In some embodiments, the point clouds in the point cloud sequence are point clouds containing noise, and the at least one reconstructed point cloud is at least one denoised point cloud.

[0117] In an embodiment of the present application, the point cloud sequence input to the encoder can be a noisy radar point cloud sequence. Through the encoding and decoding process of the embodiment of the present application, the decoder can output a denoised reconstructed point cloud without using an additional denoising method. As shown in Figure 8, the radar point cloud sequence with different degrees of noise added is projected into a depth map sequence and input into the initial network of the encoder for overfitting training. The trained network parameters are encoded and decoded, and the reconstructed point cloud output by the decoder based on the decoded network parameters has a denoising effect. Compared with the reconstructed point cloud of G-PCC with the same degree of distortion and different filter combinations, better performance evaluation indicators can be obtained. In some embodiments, Gaussian noise and random noise with different degrees of distortion are used, and CD (Chamfer distance) is used as the evaluation indicator. The experimental results of denoising are shown in Table 2. Compared with the reconstructed point cloud of G-PCC with the same degree of distortion and different filter combinations, the CD value of the embodiment of the present application performs better, indicating that the denoising effect is better.

[0118] Table 2

[0119] In some embodiments, the original lidar point cloud sequence can be geometrically lossy compressed and decoded using the encoding and decoding method of the embodiments of the present application, and the obtained reconstructed point cloud can be used in downstream visual tasks such as target detection and semantic segmentation to achieve better performance.

[0120] In some embodiments, a pre-trained semantic segmentation network is used to perform semantic segmentation on at least one reconstructed point cloud, and a semantic segmentation result corresponding to the at least one reconstructed point cloud is determined.

[0121] Exemplarily, as shown in FIG9 , the radar point cloud sequence is projected into a depth map sequence and then input into the initial network of the encoder for overfitting training. The trained network parameters are encoded and decoded. The decoder performs semantic segmentation on the reconstructed point cloud output based on the decoded network parameters, and the semantic segmentation result corresponding to the reconstructed point cloud can be obtained.

[0122] In some embodiments, a pre-trained object detection network can also be used to perform semantic segmentation on at least one reconstructed point cloud to determine the object detection result corresponding to the at least one reconstructed point cloud. The reconstructed point cloud obtained by the encoding and decoding method of the embodiment of the present application can also be applied to other visual task processing, and the specific selection depends on the actual situation and is not limited by the embodiment of the present application.

[0123] Exemplarily, the reconstructed point cloud output after the compression process shown in Figure 7 is sent to the pre-trained semantic segmentation network. Compared with the G-PCC reconstructed point cloud with the same degree of distortion, the reconstructed point cloud obtained by the encoding and decoding method of the embodiment of the present application can achieve better performance evaluation indicators. As shown in Figures 10A and 10B, the performance of the embodiment of the present application (NeRI) and G-PCC under the same geometric distortion is compared in terms of the mean overall semantic classes of class Intersection Over Union per-class intersection over union (mIoU) and overall accuracy (OA). It can be seen that under the same peak signal-to-noise ratio (PSNR), the mIoU and OA values ​​of the embodiment of the present application are higher. This data shows that the encoding and decoding reconstruction quality of the embodiment of the present application is higher, and it has obvious advantages in actual visual task processing compared to the current G-PCC encoding and decoding.

[0124] The embodiment of the present application provides a decoder 1, as shown in FIG11 , comprising:

[0125] The decoding part 11 is configured to decode the encoded information corresponding to the point cloud sequence in the code stream and determine the point cloud representation network parameters; the point cloud representation network parameters are determined by the encoder updating the network parameters of the initial network based on the temporal information and spatial information corresponding to the point cloud sequence and the depth map sequence corresponding to the point cloud sequence;

[0126] A network parameter determination section 12 is configured to determine a point cloud representation network based on the point cloud representation network parameters;

[0127] The image reconstruction part 13 is configured to use the point cloud representation network to determine at least one reconstructed depth image according to the time information and spatial information corresponding to the point cloud sequence;

[0128] The point cloud reconstruction part 14 is configured to determine at least one reconstructed point cloud corresponding to the point cloud sequence based on the at least one reconstructed depth image.

[0129] In some embodiments, the time information corresponding to the point cloud sequence includes: the time information corresponding to each point cloud in the point cloud sequence; the spatial information corresponding to the point cloud sequence includes: the spatial information corresponding to each point cloud; the image reconstruction part 13 is also configured to perform spatiotemporal position encoding on at least one time information and at least one spatial information corresponding to at least one point cloud in the point cloud sequence, and determine at least one spatiotemporal encoding vector; use the point cloud representation network to perform restoration and reconstruction based on the at least one spatiotemporal encoding vector to determine the at least one reconstructed depth image.

[0130] In some embodiments, the image reconstruction part 13 is further configured to, for each point cloud in the at least one point cloud, position encode the time information corresponding to each point cloud to determine the time encoding vector corresponding to each point cloud; position encode the spatial information corresponding to each point cloud to determine the space encoding vector corresponding to each point cloud; merge the time encoding vector with the space encoding vector to determine the space-time encoding vector corresponding to each point cloud, thereby determining the at least one space-time encoding vector.

[0131] In some embodiments, the spatial information includes translation information and rotation information; the network parameter determination part 12 is further configured to encode the translation information to determine a translation coding vector; encode the rotation information to determine a rotation coding vector; and determine the translation coding vector and the rotation coding vector as the spatial coding vector.

[0132] In some embodiments, the network parameter determination part 12 is further configured to determine an initial network, and update the network parameters of the initial network using the point cloud representation network parameters to determine the point cloud representation network.

[0133] In some embodiments, the initial network includes: a multi-layer perceptron and a multi-layer cascaded convolutional layer, the multi-layer perceptron is used to express the initial depth map signal; the multi-layer cascaded convolutional layer is used to perform convolution recovery based on the initial depth map signal to determine the reconstructed depth image.

[0134] In some embodiments, the point cloud reconstruction part 14 is further configured to perform coordinate inverse mapping on the at least one reconstructed depth image to determine the at least one reconstructed point cloud.

[0135] In some embodiments, the point clouds in the point cloud sequence are point clouds containing noise, and the at least one reconstructed point cloud is at least one denoised point cloud.

[0136] In some embodiments, the decoder 1 further includes a semantic segmentation part; the semantic segmentation part is configured to use a pre-trained semantic segmentation network to perform semantic segmentation on the at least one reconstructed point cloud and determine a semantic segmentation result corresponding to the at least one reconstructed point cloud.

[0137] In some embodiments, the decoding part 11 is further configured to receive the time information corresponding to the point cloud sequence, the spatial information corresponding to the point cloud sequence, and the code stream sent by the encoder.

[0138] The embodiment of the present application provides an encoder 2, as shown in FIG12 , including:

[0139] A mapping part 21 is configured to perform coordinate transformation mapping on the point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence;

[0140] The spatiotemporal coding section 22 is configured to perform spatiotemporal position coding on the time information and spatial information corresponding to the point cloud sequence, and determine a spatiotemporal coding vector sequence corresponding to the depth map sequence; the time information represents the acquisition time corresponding to the point cloud sequence; the spatial information represents the spatial motion information of the acquisition device of the point cloud sequence;

[0141] A parameter updating part 23 is configured to update the network parameters of the initial network using the depth map sequence and the spatiotemporal coding vector sequence to determine the point cloud representation network parameters;

[0142] The encoding part 24 is configured to encode the point cloud representation network parameters and determine the encoding information corresponding to the point cloud sequence.

[0143] In some embodiments, the time information corresponding to the point cloud sequence includes: the time information corresponding to each point cloud in the point cloud sequence; the spatial information corresponding to the point cloud sequence includes: the spatiotemporal coding part 22 is further configured to, for each point cloud in the point cloud sequence, position encode the time information corresponding to each point cloud, and determine the time coding vector corresponding to each depth map; position encode the spatial information corresponding to each point cloud, and determine the spatial coding vector corresponding to each depth map; merge the time coding vector with the spatial coding vector, and determine the spatiotemporal coding vector corresponding to each depth map, thereby determining a spatiotemporal coding vector sequence.

[0144] In some embodiments, the spatial information includes translation information and rotation information; the spatiotemporal coding part 22 is further configured to encode the translation information to determine a translation coding vector; encode the rotation information to determine a rotation coding vector; and determine the translation coding vector and the rotation coding vector as the spatial coding vector.

[0145] In some embodiments, the initial network includes: a multi-layer perceptron and a multi-layer cascaded convolutional layer, the multi-layer perceptron is used to express the initial depth map signal; the multi-layer cascaded convolutional layer is used to perform convolution recovery based on the initial depth map signal to determine the reconstructed depth image.

[0146] In some embodiments, the parameter updating part 23 is further configured to reconstruct each depth map in the depth map sequence through the initial network according to the spatiotemporal coding vector corresponding to each depth map in the spatiotemporal coding vector sequence, and determine the reconstructed depth image corresponding to each depth map; based on the reconstructed depth image and each depth map, iteratively update the network parameters of the initial network until the preset update conditions are met, and then determine the point cloud representation network parameters based on the currently updated network parameters.

[0147] In some embodiments, the parameter updating part 23 is further configured to perform pruning and / or network parameter quantization on the initial network based on the currently updated network parameters to determine the point cloud representation network parameters.

[0148] In some embodiments, the encoder 2 also includes a sending part, which is configured to generate a code stream based on the network parameter encoding information corresponding to the point cloud sequence; and send the time information corresponding to the point cloud sequence, the spatial information corresponding to the point cloud sequence, and the code stream to the decoder.

[0149] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0150] In some embodiments, the present application also provides a decoder. Figure 13 is a schematic diagram of an optional structure of a decoder 3 provided in the present application. As shown in Figure 13, the decoder 3 includes a first memory 32 and a first processor 33. The first memory 32 and the first processor 33 are connected via a first communication bus 34. The first memory 32 is used to store executable instructions. The first processor 33 is used to implement the decoding method provided in the present application when executing the executable instructions stored in the first memory 32.

[0151] In some embodiments, the present application also provides an encoder. Figure 14 is a schematic diagram of an optional structure of an encoder 4 provided in the present application. As shown in Figure 14, encoder 4 includes a second memory 42 and a second processor 43. The second memory 42 and the second processor 43 are connected via a second communication bus 44. The second memory 42 is used to store executable instructions. The second processor 43 is used to implement the encoding method provided in the present application when executing the executable instructions stored in the second memory 42.

[0152] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a first processor, the first processor will be caused to execute any one of the decoding methods provided in the embodiments of the present application; or, when the executable instructions are executed by a second processor, the second processor will be caused to execute any one of the encoding methods provided in the embodiments of the present application.

[0153] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0154] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0155] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0156] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0157] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0158] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0159] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0161] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application. Industrial Applicability

[0162] Embodiments of the present application provide a coding and decoding method, decoder, encoder, and computer-readable storage medium. The decoder determines a point cloud representation network based on point cloud representation network parameters transmitted by the encoder, reconstructs at least one depth image using the point cloud representation network and temporal and spatial information corresponding to at least one point cloud, and then recovers at least one reconstructed point cloud based on the at least one depth image. This implements decoding and reconstruction of a depth image represented by implicit neural network parameters using a neural network based on spatiotemporal position coding. This method can more fully leverage the advantages of neural networks, significantly reduce computational complexity, improve decoding efficiency, and enhance the quality of decoded reconstruction. Furthermore, high-frequency components of the point cloud representation network are derived based on the temporal and spatial information corresponding to at least one point cloud, which can enhance the representation effect of the neural network, further improve reconstruction quality, and thus enhance decoding performance. The encoder converts the point cloud into a depth map, making its spatial arrangement more efficient and compact. Through spatiotemporal position coding, the depth image is represented using an implicit neural network, further enhancing the representation efficiency of the point cloud sequence. This reduces computational complexity and improves coding efficiency. Moreover, the encoder of the embodiment of the present application not only performs position encoding on the time information, but also combines the spatial posture information of the sensor (such as translation and rotation) to perform the splicing of the encoding vectors, which are used together as the input of the implicit neural network, thereby achieving a better representation effect, thereby improving the encoding quality and further improving the encoding performance.

Claims

1. A decoding method, comprising: Decode the coded information corresponding to the point cloud sequence in the code stream to determine the point cloud representation network parameters; The point cloud representation network parameters are determined by the encoder updating the network parameters of the initial network based on the time information and spatial information corresponding to the point cloud sequence and the depth map sequence corresponding to the point cloud sequence; Determining a point cloud representation network based on the point cloud representation network parameters; Determine at least one reconstructed depth image using the point cloud representation network according to time information and spatial information corresponding to the point cloud sequence; The time information represents the acquisition time corresponding to the point cloud sequence; The spatial information represents the spatial motion information of the acquisition device of the point cloud sequence; At least one reconstructed point cloud corresponding to the point cloud sequence is determined based on the at least one reconstructed depth image.

2. The method according to claim 1, wherein: The time information corresponding to the point cloud sequence includes: the time information corresponding to each point cloud in the point cloud sequence; the spatial information corresponding to the point cloud sequence includes: the spatial information corresponding to each point cloud; the using the point cloud representation network to determine at least one reconstructed depth image according to the time information and spatial information corresponding to the point cloud sequence, includes: Performing spatiotemporal position coding on at least one time information and at least one space information corresponding to at least one point cloud in the point cloud sequence to determine at least one spatiotemporal coding vector; The point cloud representation network is utilized to perform restoration and reconstruction according to the at least one spatiotemporal coding vector to determine the at least one reconstructed depth image.

3. The method according to claim 2, wherein: The step of performing spatiotemporal position coding on at least one time information and at least one spatial information corresponding to at least one point cloud in the point cloud sequence to determine at least one spatiotemporal coding vector includes: For each point cloud in the at least one point cloud, position-encode the time information corresponding to each point cloud, and determine a time encoding vector corresponding to each point cloud; Performing position encoding on the spatial information corresponding to each point cloud to determine the spatial encoding vector corresponding to each point cloud; The time coding vector and the space coding vector are combined to determine the time-space coding vector corresponding to each point cloud, thereby determining the at least one time-space coding vector.

4. The method according to claim 3, wherein: The spatial information includes translation information and rotation information; the position encoding of the spatial information corresponding to each point cloud to determine the spatial encoding vector corresponding to each point cloud includes: Encoding the translation information to determine a translation coding vector; Encoding the rotation information to determine a rotation encoding vector; The translation encoding vector and the rotation encoding vector are determined as the spatial encoding vector.

5. The method according to any one of claims 1 to 4, wherein: The step of determining a point cloud representation network based on the point cloud representation network parameters includes: An initial network is determined, and network parameters of the initial network are updated using the point cloud representation network parameters to determine the point cloud representation network.

6. The method according to claim 5, wherein: The initial network includes: a multi-layer perceptron and a multi-layer cascaded convolutional layer, the multi-layer perceptron is used to express an initial depth map signal; the multi-layer cascaded convolutional layer is used to perform convolution recovery based on the initial depth map signal to determine a reconstructed depth image.

7. The method according to any one of claims 1 to 4, or claim 6, wherein: The step of determining, based on the at least one reconstructed depth image, at least one reconstructed point cloud corresponding to the point cloud sequence comprises: Coordinate inverse mapping is performed on the at least one reconstructed depth image to determine the at least one reconstructed point cloud.

8. The method according to any one of claims 1 to 4, or claim 6, wherein: The point clouds in the point cloud sequence are point clouds containing noise, and the at least one reconstructed point cloud is at least one denoised point cloud.

9. The method according to any one of claims 1 to 4, or claim 6, wherein: The method further comprises: Using a pre-trained semantic segmentation network, semantic segmentation is performed on the at least one reconstructed point cloud to determine a semantic segmentation result corresponding to the at least one reconstructed point cloud.

10. The method according to any one of claims 1 to 4, or claim 6, wherein: The method further comprises: Receive time information corresponding to the point cloud sequence, spatial information corresponding to the point cloud sequence, and the code stream sent by an encoder.

11. A coding method, comprising: Performing coordinate transformation mapping on the point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence; Performing spatiotemporal position encoding on the time information and spatial information corresponding to the point cloud sequence to determine a spatiotemporal encoding vector sequence corresponding to the depth map sequence; the time information represents the acquisition time corresponding to the point cloud sequence; The spatial information represents the spatial motion information of the acquisition device of the point cloud sequence; Using the depth map sequence and the spatiotemporal coding vector sequence, updating the network parameters of the initial network to determine the point cloud representation network parameters; The point cloud representation network parameters are encoded to determine encoding information corresponding to the point cloud sequence.

12. The encoding method according to claim 11, wherein: The time information corresponding to the point cloud sequence includes: the time information corresponding to each point cloud in the point cloud sequence; the spatial information corresponding to the point cloud sequence includes: the spatial information corresponding to each point cloud; the time information and spatial information corresponding to the point cloud sequence are temporally and spatially encoded to determine the temporal and spatial encoding vector sequence corresponding to the depth map sequence, including: For each point cloud in the point cloud sequence, position encoding is performed on the time information corresponding to each point cloud, and a time encoding vector corresponding to each depth map is determined; Performing position encoding on the spatial information corresponding to each point cloud to determine a spatial encoding vector corresponding to each depth map; The temporal coding vector and the spatial coding vector are combined to determine the spatiotemporal coding vector corresponding to each depth map, thereby determining a spatiotemporal coding vector sequence.

13. The method according to claim 12, wherein: The spatial information includes translation information and rotation information; and the step of performing position encoding on the spatial information corresponding to each depth map to determine the spatial encoding vector corresponding to each depth map includes: Encoding the translation information to determine a translation coding vector; Encoding the rotation information to determine a rotation encoding vector; The translation encoding vector and the rotation encoding vector are determined as the spatial encoding vector.

14. The method according to any one of claims 11 to 13, wherein: The initial network includes: a multi-layer perceptron and a multi-layer cascaded convolutional layer, the multi-layer perceptron is used to express an initial depth map signal; the multi-layer cascaded convolutional layer is used to perform convolution recovery based on the initial depth map signal to determine a reconstructed depth image.

15. The method according to any one of claims 11 to 13, wherein: The method of updating the network parameters of the initial network by using the depth map sequence and the spatiotemporal coding vector sequence to determine the point cloud representation network parameters includes: Reconstructing each depth map in the depth map sequence according to the spatiotemporal coding vector corresponding to each depth map in the spatiotemporal coding vector sequence through the initial network, and determining a reconstructed depth image corresponding to each depth map; Based on the reconstructed depth image and each depth map, the network parameters of the initial network are iteratively updated until a preset update condition is reached, and then the point cloud representation network parameters are determined based on the currently updated network parameters.

16. The method according to claim 15, wherein: The step of determining the point cloud representation network parameters based on the currently updated network parameters includes: Based on the currently updated network parameters, the initial network is pruned and / or the network parameters are quantized to determine the point cloud representation network parameters.

17. The method according to any one of claims 1 to 3, or claim 6, wherein: The method further comprises: Generate a bitstream based on network parameter encoding information corresponding to the point cloud sequence; The time information corresponding to the point cloud sequence, the space information corresponding to the point cloud sequence and the code stream are sent to a decoder.

18. A decoder comprising: The decoding part is configured to decode the encoded information corresponding to the point cloud sequence in the code stream and determine the point cloud representation network parameters; The point cloud representation network parameters are the encoder based on the temporal information and spatial information corresponding to the point cloud sequence, and the The depth map sequence corresponding to the point cloud sequence is determined by updating the network parameters of the initial network; A network parameter determination part, configured to determine a point cloud representation network based on the point cloud representation network parameters; An image reconstruction part is configured to use the point cloud representation network to determine at least one reconstructed depth image according to the time information and spatial information corresponding to the point cloud sequence; The point cloud reconstruction part is configured to determine at least one reconstructed point cloud corresponding to the point cloud sequence based on the at least one reconstructed depth image.

19. An encoder comprising: A mapping part is configured to perform coordinate transformation mapping on the point cloud sequence to determine a depth map sequence corresponding to the point cloud sequence; The spatiotemporal coding part is configured to perform spatiotemporal position coding on the time information and spatial information corresponding to the point cloud sequence to determine the spatiotemporal coding vector sequence corresponding to the depth map sequence; the time information represents the acquisition time corresponding to the point cloud sequence; The spatial information represents the spatial motion information of the acquisition device of the point cloud sequence; A parameter updating part is configured to update the network parameters of the initial network using the depth map sequence and the spatiotemporal coding vector sequence to determine the point cloud representation network parameters; The encoding part is configured to encode the point cloud representation network parameters and determine the encoding information corresponding to the point cloud sequence.

20. A code stream, comprising: The code stream is generated by bit encoding according to the encoding information; wherein the encoding information at least includes: encoding information corresponding to the point cloud sequence; the encoding information corresponding to the point cloud sequence is obtained by encoding the point cloud representation network parameters; the point cloud representation network parameters are determined by updating the network parameters of the initial network using the depth map sequence corresponding to the point cloud sequence, and the time information and space information corresponding to the point cloud sequence.

21. A decoder comprising: A first memory configured to store executable instructions; The first processor is configured to implement the method according to any one of claims 1 to 10 when executing the executable instructions stored in the first memory.

22. An encoder comprising: a second memory configured to store executable instructions; The second processor is configured to implement the method according to any one of claims 11 to 17 when executing the executable instructions stored in the second memory.

23. A computer-readable storage medium storing executable instructions for causing a first processor to execute the method according to any one of claims 1 to 10, or for causing a second processor to execute the method according to any one of claims 11 to 17.

Images