A method and a target processing point cloud data model

Abstract

This application provides a method for processing point cloud data and a target point cloud data processing model. The method includes: acquiring point cloud data to be compressed, wherein the point cloud data to be compressed is obtained by point cloud data collection of a target scene or target object; inputting the point cloud data to be compressed into a compression model, and obtaining compressed point cloud data through the compression model, wherein the compression model includes: a multi-layer feature extraction module, each feature extraction module further including a local feature extraction module and a global feature extraction module connected to the local feature extraction module, the global feature extraction module including at least a self-attention module and a feedforward neural network; and performing image processing based on the compressed point cloud data. Through some embodiments of this application, point cloud data compression can be achieved, and global features of the point cloud data can be obtained, thereby improving the accuracy of subsequent image processing.

Description

Technical Field

[0001] This application relates to the field of point cloud processing, specifically to a method for processing point cloud data and a target point cloud data processing model. Background Technology

[0002] 3D point clouds are an important form of digital representation of the real world. The massive data volume of 3D point clouds poses significant challenges to data storage and transmission; therefore, point cloud data is usually compressed before subsequent steps. Related technologies have proposed compression for static point clouds, utilizing local features of the point cloud data for compression. However, local features cannot accurately represent the true characteristics of the point cloud data, leading to reduced accuracy in subsequent recognition and reconstruction steps.

[0003] Therefore, how to accurately represent the features of point cloud data during the process of compressing point cloud data using local features has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method for processing point cloud data and a target point cloud data processing model. Through some embodiments of this application, at least point cloud data compression can be achieved, and global features of point cloud data can be obtained, thereby improving the accuracy of subsequent image processing.

[0005] In a first aspect, embodiments of this application provide a method for processing point cloud data. The method includes: acquiring point cloud data to be compressed, wherein the point cloud data to be compressed is obtained by collecting point cloud data of a target scene or target object; inputting the point cloud data to be compressed into a compression model, and obtaining compressed point cloud data through the compression model, wherein the compression model includes: a multi-layer feature extraction module, each feature extraction module further including a local feature extraction module and a global feature extraction module connected to the local feature extraction module, the global feature extraction module including at least a self-attention module and a feedforward neural network; and performing image processing based on the compressed point cloud data.

[0006] Therefore, unlike the methods in related technologies that compress point cloud data using only local features, the embodiments of this application use a global feature extraction module connected to the local feature extraction module to extract global depth features from the point cloud data, thereby improving the accuracy of subsequent image processing.

[0007] In conjunction with the first aspect, in one embodiment of this application, the global feature extraction module further includes: a first normalization module and a second normalization module; wherein, the first normalization module is configured to: input the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data into the first normalization module; perform a summation operation on the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data through the first normalization module to obtain a first summed feature; perform a normalization operation on the first summed feature to obtain a first normalized feature; the second normalization module is configured to: input the output of the feedforward neural network and the first normalized feature into the second normalization module; perform a summation operation on the output of the feedforward neural network and the first normalized feature through the second normalization module to obtain a second summed feature; perform a normalization operation on the second summed feature to obtain a downsampled global feature.

[0008] Therefore, in this embodiment of the application, the downsampled local features and the features corresponding to self-attention are fused by the first normalization module and the second normalization module in the global feature extraction module, so as to obtain the global features of point cloud data and improve the accuracy of subsequent downstream tasks.

[0009] In conjunction with the first aspect, in one embodiment of this application, obtaining compressed point cloud data through the compression model includes: inputting the output of the (i-1)th global feature extraction module into the i-th local feature extraction module for local feature extraction to obtain the i-th downsampled local feature; inputting the output of the (i-1)th global feature extraction module and the i-th downsampled local feature into the i-th global feature extraction module to obtain the i-th downsampled global feature; wherein, i is an integer greater than or equal to 1, the i-th downsampled global feature is the data obtained after compressing the point cloud data at a target ratio, and the target ratio is determined by all downsampled convolutional kernels in the i-th global feature extraction module; repeating the above steps until the compression bitrate of the point cloud data meets the requirements to obtain the target downsampled global feature; and inputting the target downsampled global feature into a multilayer perceptron to obtain compressed point cloud data.

[0010] Therefore, the embodiments of this application calculate global features based on the Transformer architecture, which can improve the accuracy of subsequent reconstruction processes; by obtaining the features extracted by all previous modules in the currently performing calculation module, the features of point cloud data can be transmitted to the deep network, thereby improving the accuracy of feature extraction.

[0011] In conjunction with the first aspect, in one embodiment of this application, before performing image processing based on the compressed point cloud data, the method further includes: inputting the compressed point cloud data into a priori encoder, performing upsampling and downsampling operations on the compressed point cloud data through the priori encoder to obtain priori point cloud data features; obtaining Gaussian distribution parameters based on the priori point cloud data features, wherein the Gaussian distribution parameters include mean and variance; and performing arithmetic encoding and decoding operations on the compressed point cloud data based on the Gaussian distribution parameters to obtain target compressed point cloud data. The image processing based on the compressed point cloud data includes: performing image processing based on the target compressed point cloud data.

[0012] Therefore, by performing upsampling, downsampling, and arithmetic encoding / decoding operations on the compressed point cloud data, this embodiment of the application can extract more representative global features, thereby obtaining the features of the entire region and improving the accuracy of the model in image processing of the target compressed point cloud data.

[0013] In conjunction with the first aspect, in one embodiment of this application, the step of performing upsampling and downsampling operations on the compressed point cloud data by the super-prior encoder to obtain super-prior point cloud data features includes: performing downsampling operations on the compressed point cloud data to obtain super-prior downsampled global features; and performing upsampling operations on the super-prior downsampled global features to obtain super-prior point cloud data features.

[0014] Therefore, by upsampling and downsampling the compressed point cloud data, this embodiment of the application can extract more representative global features, thereby obtaining the features of the entire region.

[0015] In conjunction with the first aspect, in one embodiment of this application, the super-prior encoder includes a multilayer perceptron and a positionless global feature extraction module; the step of downsampling the compressed point cloud data to obtain super-prior downsampled global features includes: inputting the output of the (i-1)th positionless global feature extraction module into the i-th multilayer perceptron to obtain the i-th perceptron feature; inputting the i-th perceptron feature and the spatial coordinates into the i-th positionless global feature extraction module for global feature extraction to obtain the i-th super-prior downsampled global feature; repeating the above steps until the prior compression bitrate of the point cloud data meets the requirements to obtain the super-prior downsampled global feature.

[0016] Therefore, this embodiment of the application can further compress the point cloud data by downsampling the compressed point cloud data. Through the combination of multilayer perceptron and positionless global feature extraction module, it can extract deep global features of the point cloud data, thereby improving the accuracy of the model in the subsequent recognition and reconstruction process.

[0017] In conjunction with the first aspect, in one embodiment of this application, the image processing based on the target compressed point cloud data includes: classifying based on the target compressed point cloud data; and / or reconstructing the target scene or the target object based on the target compressed point cloud data.

[0018] Therefore, by performing image processing on the target compressed point cloud data after compression, the embodiments of this application can improve the rate-distortion performance and classification accuracy of the model.

[0019] In conjunction with the first aspect, in one embodiment of this application, reconstructing the target scene or the target object based on the target compressed point cloud data includes: inputting the target compressed point cloud data into a decoding module, and obtaining the target scene or target object through the decoding module, wherein the decoding module includes: a multi-layer point cloud decoding module, each layer of the point cloud decoding module further including a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer.

[0020] Therefore, the embodiments of this application decode the target compressed point cloud data by decoding the convolutional layer and the positionless global feature extraction module, which can use the global features of the point cloud data to reconstruct the target scene or target object, thereby improving the accuracy of reconstruction.

[0021] In conjunction with the first aspect, in one embodiment of this application, the method is applied to a target processing point cloud data model, the target processing point cloud data model comprising: a compression model, a super-prior encoder, a decoding module, and a recognition module, the target processing point cloud data model being trained using a target loss function; wherein, the target loss function is characterized by both the reconstruction result loss value and the recognition result loss value, the reconstruction result loss value being determined by the compression model, the super-prior encoder, and the decoding module, and the recognition result loss value being determined by the compression model, the super-prior encoder, and the recognition module.

[0022] Therefore, the embodiments of this application can calculate the overall loss of compression, reconstruction and recognition by using the target loss function value, so that the model can take into account the downstream recognition task during the compression and reconstruction process, thereby improving the accuracy of the recognition task.

[0023] In conjunction with the first aspect, in one embodiment of this application, before obtaining the point cloud data to be compressed, the method further includes: inputting the training data to be compressed into the point cloud data model to be trained for training to obtain the target point cloud data model, wherein the training data to be compressed is obtained by adding noise data to the original point cloud data.

[0024] Therefore, this application embodiment adds noise data to the original point cloud data during the training process, making it differentiable during quantization, thereby ensuring the accuracy of the trained target processing point cloud data in performing reconstruction and recognition tasks.

[0025] Secondly, embodiments of this application provide a target processing point cloud data model, the model comprising: a reconstruction module, used to compress and reconstruct the point cloud data to be compressed to obtain a reconstructed image, wherein the reconstruction module comprises: a compression model, used to downsample and encode the point cloud data to obtain compressed point cloud data; a super-prior encoder, used to obtain Gaussian distribution parameters based on the compressed point cloud data; a decoding module, used to upsample and decode the compressed point cloud data using the Gaussian distribution parameters to obtain the reconstructed image, the reconstructed image being a target scene image or a target object image; and a recognition module, used to classify the compressed point cloud data to obtain the type of the point cloud data.

[0026] In conjunction with the second aspect, in one embodiment of this application, the compression model consists of a multi-layer feature extraction module and a multi-layer perceptron.

[0027] In conjunction with the second aspect, in one embodiment of this application, the decoding module consists of a decoding convolutional layer and a positionless global feature extraction module.

[0028] Thirdly, embodiments of this application provide an apparatus for processing point cloud data. The apparatus includes: a data acquisition module configured to acquire point cloud data to be compressed, wherein the point cloud data to be compressed is obtained by collecting point cloud data of a target scene or target object; a data compression module configured to input the point cloud data to be compressed into a compression model and obtain compressed point cloud data through the compression model, wherein the compression model includes: a multi-layer feature extraction module, each feature extraction module further including a local feature extraction module and a global feature extraction module connected to the local feature extraction module, the global feature extraction module including at least a self-attention module and a feedforward neural network; and an image processing module configured to perform image processing based on the compressed point cloud data.

[0029] In conjunction with the third aspect, in one embodiment of this application, the global feature extraction module further includes: a first normalization module and a second normalization module; wherein, the first normalization module is configured to: input the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data into the first normalization module; perform a summation operation on the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data through the first normalization module to obtain a first summed feature; perform a normalization operation on the first summed feature to obtain a first normalized feature; the second normalization module is configured to: input the output of the feedforward neural network and the first normalized feature into the second normalization module; perform a summation operation on the output of the feedforward neural network and the first normalized feature through the second normalization module to obtain a second summed feature; perform a normalization operation on the second summed feature to obtain a downsampled global feature.

[0030] In conjunction with the third aspect, in one embodiment of this application, the data compression module is configured to: input the output of the (i-1)th global feature extraction module into the i-th local feature extraction module for local feature extraction to obtain the i-th downsampled local feature; input the output of the (i-1)th global feature extraction module and the i-th downsampled local feature into the i-th global feature extraction module to obtain the i-th downsampled global feature; wherein i is an integer greater than or equal to 1, the i-th downsampled global feature is the data obtained after compressing the point cloud data at a target ratio, the target ratio being determined by all downsampled convolutional kernels in the i-th global feature extraction module; repeat the above steps until the compression bitrate of the point cloud data meets the requirements to obtain the target downsampled global feature; input the target downsampled global feature into a multilayer perceptron to obtain compressed point cloud data.

[0031] In conjunction with the third aspect, in one embodiment of this application, the data compression module is configured to: input the compressed point cloud data into a priori encoder; perform upsampling and downsampling operations on the compressed point cloud data using the priori encoder to obtain priori point cloud data features; obtain Gaussian distribution parameters based on the priori point cloud data features, wherein the Gaussian distribution parameters include mean and variance; and perform arithmetic encoding and decoding operations on the compressed point cloud data based on the Gaussian distribution parameters to obtain target compressed point cloud data. The image processing module is configured to perform image processing based on the target compressed point cloud data.

[0032] In conjunction with the third aspect, in one embodiment of this application, the data compression module is configured to: perform a downsampling operation on the compressed point cloud data to obtain super-prior downsampled global features; and perform an upsampling operation on the super-prior downsampled global features to obtain super-prior point cloud data features.

[0033] In conjunction with the third aspect, in one embodiment of this application, the super-prior encoder includes a multilayer perceptron and a positionless global feature extraction module; the data compression module is configured to: input the output of the (i-1)th positionless global feature extraction module into the i-th multilayer perceptron to obtain the i-th perceptron feature; input the i-th perceptron feature and the spatial coordinates into the i-th positionless global feature extraction module for global feature extraction to obtain the i-th super-prior downsampled global feature; repeat the above steps until the prior compression bitrate of the point cloud data meets the requirements to obtain the super-prior downsampled global feature.

[0034] In conjunction with the third aspect, in one embodiment of this application, the image processing module is configured to: classify the target compressed point cloud data; and / or reconstruct the target scene or the target object based on the target compressed point cloud data.

[0035] In conjunction with the third aspect, in one embodiment of this application, the image processing module is configured to: input the target compressed point cloud data into the decoding module, and obtain the target scene or target object through the decoding module, wherein the decoding module includes: a multi-layer point cloud decoding module, each layer of the point cloud decoding module further including a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer.

[0036] In conjunction with the third aspect, in one embodiment of this application, the method is applied to a target processing point cloud data model, which includes a compression model, a super-prior encoder, a decoding module, and a recognition module. The target processing point cloud data model is trained using a target loss function. The target loss function is characterized by both the reconstruction result loss value and the recognition result loss value. The reconstruction result loss value is determined by the compression model, the super-prior encoder, and the decoding module, and the recognition result loss value is determined by the compression model, the super-prior encoder, and the recognition module.

[0037] In conjunction with the third aspect, in one embodiment of this application, the data acquisition module is configured to: input the training data to be compressed into the point cloud data model to be trained for training, thereby obtaining the target point cloud data model, wherein the training data to be compressed is obtained by adding noise data to the original point cloud data.

[0038] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus; the processor is connected to the memory via the bus, and the memory stores computer-readable instructions, which, when executed by the processor, are used to implement the method as described in any embodiment of the first aspect.

[0039] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed, implements the method as described in any embodiment of the first aspect. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the system composition for point cloud data processing according to an embodiment of this application;

[0041] Figure 2 This application illustrates a target point cloud processing model in an embodiment.

[0042] Figure 3 This is a flowchart illustrating a method for processing target point clouds according to an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the global feature extraction module structure shown in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a local feature extraction module shown in an embodiment of this application;

[0045] Figure 6 This is a structural diagram of a device for processing point cloud data, as shown in an embodiment of this application.

[0046] Figure 7 This is a structural diagram of an electronic device as shown in an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the protection scope of this application.

[0048] This application can be applied to scenarios involving the compression of point cloud data and image processing of the compressed point cloud data. To address the problems in the background technology, in some embodiments of this application, a multi-layer feature extraction module is used to compress the point cloud data to be compressed, resulting in compressed point cloud data. For example, in some embodiments of this application, the electronic device is configured to at least: input the point cloud data to be compressed into multiple local feature extraction modules to obtain multiple downsampled local features, and then input the multiple downsampled local features and the position information corresponding to the multiple downsampled local features into a global feature extraction module for compression to obtain compressed point cloud data.

[0049] It should be noted that the point cloud data to be compressed is obtained by collecting point cloud data from the target scene or target object. The target scene can be any scene that has ever occurred, such as digital 3D modeling of smart cities, high-definition 3D maps for autonomous driving, and digital preservation of historical artifacts. The target object can be any object or person, such as a person, airplane, house, train, etc.

[0050] The method steps in the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0051] Figure 1 This application provides schematic diagrams of a system composition for processing point cloud data in some embodiments. The system includes a point cloud data acquisition device 110, a server 120, and a client 130. Specifically, the point cloud data acquisition device 110 acquires point cloud data of a target scene or target object to obtain point cloud data to be compressed. This compressed point cloud data is then input into the server 120, which executes a method for processing the point cloud data to obtain the type of point cloud data and an image of the target scene (or target object). Finally, the type and image are input into the client 130 for display. Relevant personnel can obtain information about the point cloud data (i.e., the type of point cloud data and the image of the target scene (or target object)) through the client 130.

[0052] In some other embodiments of this application, after obtaining the point cloud data to be compressed, the point cloud data acquisition device 110 first performs a compression operation on the point cloud data to be compressed to obtain compressed point cloud data, and then inputs the compressed point cloud data into the server 120. The server 120 obtains the type of point cloud data and the target scene (or target object) image based on the compressed point cloud data.

[0053] The static point cloud compression schemes proposed in related technologies utilize local features of point cloud data for compression. However, local features cannot accurately represent the true features of point cloud data, leading to a decrease in the accuracy of subsequent recognition and reconstruction steps. In contrast, the feature extraction modules at each layer of the embodiments of this application further obtain global features of the point cloud data based on the extracted local features.

[0054] The target point cloud data model used to implement the point cloud data processing method in the embodiments of this application will be described below.

[0055] In one embodiment of this application, such as Figure 2 As shown, the target point cloud data processing model (i.e., a trained neural network model with point cloud data compression and certain image processing capabilities) includes a compression model 210. The compression model 210 is used to compress the point cloud data to be compressed after acquiring the point cloud data to be compressed, so as to obtain the compressed point cloud data.

[0056] Specifically, the compression model 210 includes a multi-layer feature extraction module (i.e., a first feature extraction module 211, a second feature extraction module 212, a third feature extraction module 213, and a fourth feature extraction module 214). It can be understood that in practical applications, the multi-layer feature extraction module may include, for example... Figure 2 The perceptron has four layers (or more or fewer), and each layer's feature extraction module further includes a local feature extraction module and a global feature extraction module connected to the local feature extraction module. A multilayer perceptron 215 is connected after all the layer feature extraction modules. Based on common knowledge, the global feature extraction module includes a self-attention module, a feedforward neural network, a first normalization module, and a second normalization module. The multilayer perceptron 215 will not be described in detail here.

[0057] The working process of compression model 210 is described below as an example. For instance, the input is the three-dimensional coordinates of the point cloud data to be compressed (represented as N×3, where N is the number of point cloud data and 3 is the three spatial coordinates of the point cloud). The second feature extraction module 212 outputs point cloud data of size N / 4, the third feature extraction module 213 outputs point cloud data of size N / 16, the fourth feature extraction module 214 outputs point cloud data of size N / 16, and the multilayer perceptron 215 outputs compressed point cloud data of size 1×256, for a total of 256 point cloud data. It can be understood that the output point cloud data is compressed compared to the input point cloud data.

[0058] It should be noted that the global feature extraction module is built on the Transformer framework.

[0059] It is understandable that the obtained compressed point cloud data (i.e., the point cloud data output by the multilayer perceptron) can be stored in a database. Since the output point cloud data is significantly reduced compared to the input point cloud data, storage space can be saved. In some embodiments of this application, operations such as target recognition or scene reconstruction based on the obtained compressed point cloud data are also provided. For example, if the target recognition and scene reconstruction devices are not located in the same place as the point cloud data compression device, the compressed data can be transmitted to the corresponding processing device via a network. If the target recognition and scene reconstruction functions are located on the same device as the point cloud compression functions, the program modules corresponding to target recognition and scene reconstruction can read the compressed data from the memory for further processing.

[0060] In other words, in one embodiment of this application, in addition to setting a point cloud data compression module, a reconstruction module is also provided. This reconstruction module is used to reconstruct the scene based on the compressed point cloud data. Figure 2 As shown, in some embodiments of this application, the reconstruction module includes a hyper-prior encoder 220 and a decoding module 230, wherein the hyper-prior encoder is used to obtain Gaussian distribution parameters based on the compressed point cloud data, and the decoding module is used to upsample and decode the compressed point cloud data using the Gaussian distribution parameters to obtain a reconstructed image.

[0061] Specifically, such as Figure 2 As shown, the super prior encoder 220 includes a multi-layer prior feature extraction module, and each layer of prior feature extraction module includes a multi-layer perceptron and a positionless global feature extraction module.

[0062] The working process of the super-prior encoder 220 is illustrated below. During the downsampling of the compressed point cloud data, the multi-layer prior feature extraction module includes a first-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222), a second-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222), and a third-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222). During the upsampling of the point cloud data output by the third-layer prior feature extraction module, the multi-layer prior feature extraction module includes a fourth-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222), a fifth-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222), and a sixth-layer prior feature extraction module (i.e., multi-layer perceptron 215 and positionless global feature extraction module 222). Finally, the multi-layer perceptron 215 outputs the super-prior point cloud data features.

[0063] It should be noted that the location-free global feature extraction module is based on the Transformer framework. Unlike the Transformer framework in related technologies, which requires local features as input, the location-free global feature extraction module takes the spatial coordinates of the point cloud data and the output features of the multilayer perceptron as input. That is, the location-free global feature extraction module in this embodiment does not require local features as input.

[0064] In one embodiment of this application, the reconstruction module further includes a decoding module 230. Before reconstructing the target scene or target object using the decoding module 230, arithmetic encoding and decoding operations need to be performed on the compressed point cloud data using the Gaussian distribution parameters output by the prior encoder 220 to obtain the target compressed point cloud data. The target compressed point cloud data is then input into the decoding module 230 to obtain the reconstructed image of the target scene or target object.

[0065] Specifically, the decoding module 230 includes a multi-layer point cloud decoding module and a multi-layer perceptron 215. Each point cloud decoding module further includes a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer. That is, the multi-layer point cloud decoding module includes a first point cloud decoding module, a second point cloud decoding module, and a third point cloud decoding module. The first point cloud decoding module includes a decoding convolutional layer 235 and a positionless global feature extraction module 222; the second point cloud decoding module includes a decoding convolutional layer 235 and a positionless global feature extraction module 222; and the third point cloud decoding module includes a decoding convolutional layer 235 and a positionless global feature extraction module 222.

[0066] For example: After inputting the target compressed point cloud data into the first point cloud decoding module, the first point cloud decoding module outputs point cloud data of size N / 16, the second point cloud decoding module outputs point cloud data of size N / 16, the third point cloud decoding module outputs point cloud data of size N / 4, and the multilayer perceptron 215 restores the point cloud data of size N×3 and reconstructs the image.

[0067] In one embodiment of this application, such as Figure 2 As shown, the target point cloud data model includes an identification module 240. To overcome the shortcomings of related technologies where point cloud data compression does not consider performance for downstream tasks (e.g., classification tasks), this embodiment of the application inputs the target compressed point cloud data into the identification module to identify the type of point data.

[0068] Specifically, the recognition module 240 includes a four-layer multilayer perceptron 215. After the target compressed point cloud data is input into the first-layer multilayer perceptron 215, it undergoes layer-by-layer feature extraction and recognition to obtain the type of the point cloud data, that is, the type of the target scene or the type of the target object. For example, after inputting the target compressed point cloud data into the recognition module 240, the type of the target object is obtained as an airplane.

[0069] The foregoing described a target processing point cloud data model in an embodiment of this application. The following describes the process of training the target processing point cloud data model.

[0070] It should be noted that the structure of the point cloud data model to be trained and the target point cloud data model obtained after training are the same.

[0071] In related technologies, during image encoding, the derivative of the piecewise function at the boundary points is 0, and there is no derivative during backpropagation at the boundary points, thus often resulting in non-differentiability. To address this issue, this application employs a probabilistic model with uniform noise instead of quantization. Specifically, during training, noise data in the range (-0.5, 0.5) is added to the original point cloud data to obtain the training data to be compressed, making the quantization process differentiable. The expression is shown in formula (1):

[0072]

[0073] Where y represents the original point cloud data, This represents the training data to be compressed, and μ represents the noise data.

[0074] The minimum average bitrate achievable by the compression and decoding modules using their respective shared entropy models is given by the Shannon cross-entropy between the two distributions, as shown in Equation (2) below:

[0075]

[0076] in, express The probability density function value, where R represents the compression bitrate. This represents the point cloud data corresponding to the reconstructed image. This represents the Shannon cross-entropy.

[0077] The VAE structure used in sample image compression performs conditional probability estimation, including structures such as the compression model and the super-prior encoder. In the super-prior encoder, the hidden layer representation learns a super-prior information (i.e., Gaussian distribution parameters) through convolutional downsampling. These Gaussian distribution parameters are used by the decoding module to assist in rate estimation.

[0078] Therefore, this application embodiment adds noise data to the original point cloud data during the training process, making it differentiable during quantization, thereby ensuring the accuracy of the trained target processing point cloud data in performing reconstruction and recognition tasks.

[0079] In related technologies, the training process for point cloud data models usually involves training reconstruction and recognition models separately without considering the performance for downstream tasks. This results in the reconstruction model focusing only on compression and reconstruction processes, leading to reduced recognition performance.

[0080] To address the aforementioned issues, this application embodiment trains the compression, reconstruction, and recognition processes together during model training, thereby improving the performance of downstream tasks. Since the compression, reconstruction, and recognition processes are trained together, this application embodiment uses the target loss function as the overall loss for the point cloud data processing model to be trained.

[0081] In other words, the target processing point cloud data model is trained using a target loss function. This target loss function is characterized by both the reconstruction result loss and the recognition result loss. The reconstruction result loss is determined by the compression model, the hyper-prior encoder, and the decoding module, while the recognition result loss is determined by the compression model, the hyper-prior encoder, and the recognition module.

[0082] Specifically, since the arithmetic encoding and decoding stages of the compressed point cloud data using Gaussian distribution parameters both require decoding the occurrence probability or cumulative probability distribution (CDF) of points, this CDF needs to be transmitted to the decoding end for correct entropy decoding. Therefore, the super-prior encoder needs to first compress the probability or cumulative probability distribution into z, then quantize and entropy encode z and transmit it to the decoding end of the super-prior encoder. The decoding end then decodes and learns the modeling parameters of the latent representation y. After obtaining the modeling distribution of the latent representation y through the super-prior encoder, it is modeled and the quantized data is then processed... Entropy encoding yields a compressed bitstream file, while entropy decoding yields... Then decode the entropy result The image is input into the decoding module to obtain the final reconstructed image. For the optimization of the encoding and decoding parameters of the overall model, the loss function of the reconstruction result is shown in the following formulas (2) and (3):

[0083]

[0084] L E =λ*D+R (3)

[0085] Where R represents the compression bitrate of the compression model, and D represents the distortion between the reconstructed image and the sample image. express The probability density function value, This represents the point cloud data corresponding to the reconstructed image. L represents the Shannon cross-entropy. E λ represents the loss value of the reconstruction result, and λ represents the coefficient.

[0086] Since the embodiment of this application trains the recognition model together with the compression model, the decoding module and the super prior encoder, that is, while compressing the feature domain, it also completes the point cloud classification task for machine vision perception, the target loss function is as shown in the following formulas (4) and (5):

[0087]

[0088] L = L E +μH (5)

[0089] Where H represents the loss value of the recognition result, This represents the point cloud data corresponding to the reconstructed image. express The corresponding probability value, express The amount of information, L E L represents the loss value resulting from the reconstruction, and L represents the target loss value (i.e., the overall loss value).

[0090] Therefore, the embodiments of this application can calculate the overall loss of compression, reconstruction and recognition by using the target loss function value, so that the model can take into account the downstream recognition task during the compression and reconstruction process, thereby improving the accuracy of the recognition task.

[0091] The training process of the model in the embodiments of this application has been described above. The following will describe a method for processing point cloud data in the embodiments of this application.

[0092] At least in order to solve the problems in the background technology, such as Figure 3 As shown in the figure, this application provides a method for processing point cloud data, including:

[0093] S310: Acquire the point cloud data to be compressed.

[0094] In other words, it involves acquiring point cloud data of the target scene or acquiring point cloud data of the target object.

[0095] S320 inputs the point cloud data to be compressed into the compression model, and obtains the compressed point cloud data through the compression model.

[0096] It is understood that this application proposes a lossy compression framework based on Transformer, using a design based on local and global feature extraction. Specifically, it employs the K-Nearest Neighbor (KNN) algorithm for feature sampling to obtain downsampled local features, while simultaneously using Transformer to obtain downsampled global features, and stacking multiple feature extraction modules to form a compression model. This application also transmits the features output by the shallow feature extraction module to the deep feature extraction module, so that each feature extraction module uses the features output by all preceding feature extraction modules for encoding.

[0097] In one embodiment of this application, S320 includes: obtaining compressed point cloud data through a local feature extraction module, a global feature extraction module, and a multilayer perceptron.

[0098] In other words, such as Figure 2 As shown, the point cloud data to be compressed is input into the first feature extraction module 211 (including a first local feature extraction module and a first global feature extraction module connected thereto). The first local feature extraction module extracts the first downsampled local features of the point cloud data to be compressed, and the first global feature extraction module obtains the first downsampled local features and uses these local features to obtain the global features. Then, the output of the (i-1)th global feature extraction module is input into the ith local feature extraction module for local feature extraction to obtain the ith downsampled local features. Finally, the output of the (i-1)th global feature extraction module and the ith downsampled local features are input into the ith global feature extraction module to obtain the ith downsampled global features.

[0099] It is understandable that i represents the location of the local feature extraction module and the global feature extraction module. i is an integer greater than or equal to 2.

[0100] Specifically, such as Figure 2 As shown, when i=2, the output of the first global feature extraction module is input into the second local feature extraction module to extract local features and obtain the second downsampled local features. Then, the output of the first global feature extraction module and the second downsampled local features are input into the second global feature extraction module to obtain the second downsampled global features.

[0101] When i=3, the output of the second global feature extraction module and the second downsampled local features are input into the third local feature extraction module for local feature extraction to obtain the third downsampled local features. Then, the output of the second global feature extraction module, the second downsampled local features, and the third downsampled local features are input into the third global feature extraction module to obtain the third downsampled global features.

[0102] When i=4, the output of the third global feature extraction module and the third downsampled local feature are input into the fourth local feature extraction module for local feature extraction to obtain the fourth downsampled local feature. Then, the output of the third global feature extraction module, the third downsampled local feature, and the fourth downsampled local feature are input into the fourth global feature extraction module to obtain the fourth downsampled global feature.

[0103] After confirming that the compression ratio of the i-th downsampled global feature meets the target compression ratio, the target downsampled global feature is obtained and input into the multilayer perceptron 215 to obtain compressed point cloud data. For example, the compression ratio of the fourth downsampled global feature is 16 times, which meets the target compression ratio. Therefore, the fourth downsampled global feature (i.e., the target downsampled global feature) is input into the multilayer perceptron 215 to obtain compressed point cloud data.

[0104] It should be noted that the features obtained by the i-th global feature extraction module and the i-th local feature extraction module are not only the features of the (i-1)-th global feature extraction module and the (i-1)-th local feature extraction module, but also the features of all modules before the (i-1)-th global feature extraction module and the (i-1)-th local feature extraction module. That is, in the module currently performing the calculation, the features obtained are the features extracted by all the preceding modules.

[0105] Therefore, the embodiments of this application calculate global features based on the Transformer architecture, which can improve the accuracy of subsequent reconstruction processes; by obtaining the features extracted by all previous modules in the currently performing calculation module, the features of point cloud data can be transmitted to the deep network, thereby improving the accuracy of feature extraction.

[0106] Due to the unordered nature of point cloud data, it is difficult to capture the positional relationships between points. Related techniques do not effectively utilize the geometric information of point clouds. Traditional 3D convolution can only obtain local features of point cloud data through a sliding window approach, failing to capture deep global features. Since the Transformer structure can effectively capture the dependencies between long-range and short-range points, this self-attention-based method, compared to Convolutional Neural Networks (CNNs), can obtain more features between shallow and deep representations and retain more spatial information. It can directly obtain the global geometric features of point cloud data by directly calculating the relationship between any two points. Therefore, this application's embodiments transfer the Transformer to point cloud compression, enabling the compression model to better utilize global features and achieve better compression performance.

[0107] In one embodiment of this application, the global feature extraction module based on the Transformer architecture includes: a self-attention module, a feedforward neural network, a first normalization module, and a second normalization module.

[0108] In other words, taking one of the global feature extraction modules as an example, the structure of the global feature extraction module is illustrated as follows: Figure 4 As shown:

[0109] Step 1: Sum the downsampled local features, then sum them with the spatial coordinates of the point cloud data, and then input them into the self-attention module 410 for attention calculation.

[0110] Step 2: Input the output of the self-attention module 410, the downsampled local features, and the spatial coordinates of the point cloud data into the first normalization module 420. The first normalization module 420 performs a summation operation on the output of the self-attention module 410, the downsampled local features, and the spatial coordinates of the point cloud data to obtain the first summation feature. Then, the first summation feature is normalized to obtain the first normalized feature.

[0111] Step 3: Input the output of the feedforward neural network 430 and the first normalized feature into the second normalization module 440, and perform a summation operation on the output of the feedforward neural network 430 and the first normalized feature through the second normalization module 440 to obtain the second summed feature.

[0112] Step 4: Normalize the second summed feature to obtain the downsampled global feature.

[0113] Specifically, the global feature extraction module mainly uses downsampled local features to further extract global features. The downsampled local features are used as positional encoding information to calculate global attention and obtain downsampled global features. The calculation process is shown in formula (6):

[0114]

[0115] Where, x i The x represents the spatial coordinates of the point cloud data at the intermediate position. j The spatial coordinates of the surrounding point cloud data are represented by h, g, p, k, and t. The mapping function is t, l, and y. i This represents the global features of the downsampled data.

[0116] Therefore, in this embodiment of the application, the downsampled local features and the features corresponding to self-attention are fused by the first normalization module and the second normalization module in the global feature extraction module, so as to obtain the global features of point cloud data and improve the accuracy of subsequent downstream tasks.

[0117] In one embodiment of this application, the local feature extraction module employs the local feature extraction method in PointNet++, using a distance metric in the spatial domain to partition the point set into overlapping local regions. Based on this, local features (shallow features) are first extracted from the geometric structure within a small area, then the scope is expanded, and higher-level features are extracted based on these local features until the global features of the entire point set are extracted.

[0118] Understandably, PointNet++ is an extension of PointNet, adding a hierarchical structure to PointNet, enabling the network to provide higher-level features over larger areas.

[0119] Specifically, such as Figure 5 As shown, Figure 2 Each local feature extraction module includes a farthest sampling module 510, a nearest neighbor algorithm 520, and a max pooling layer 530. Specifically, taking the first local feature extraction module as an example:

[0120] Step 1: Input the point cloud data N to be compressed into the farthest sampling module 510, and select N' points through the farthest point sampling (FPS) algorithm.

[0121] Step 2: Input N' points into the nearest neighbor algorithm 520, and generate N' corresponding local regions for the points sampled by the farthest sampling module 510 through a query method (e.g., the Ball query method).

[0122] Step 3: Input the N′ corresponding local regions into the max pooling layer 530 to extract local global features and obtain downsampled local features.

[0123] It should be noted that the input to each feature extraction module is (N, (d+C)), where N represents the number of input points, d represents the coordinate dimension, and C represents the feature dimension. The output is (N', (d+C')), where N' represents the number of output points, and C' represents the new feature dimension of the output.

[0124] As a specific embodiment of this application, the compression model consists of a multi-layer local feature extraction module stack and a multi-layer global feature extraction module. Multi-layer attention feature fusion is performed using a non-local approach. The non-local structure is divided into three branches from top to bottom: query, key, and value. The calculation typically involves three steps: (1) calculating the similarity between the query and each key to obtain weights; (2) normallyizing these weights using softmax; and (3) weighted summing of the normalized weights and the corresponding values ​​for each key to obtain the final attention features.

[0125] During the encoding process, the global feature extraction module is used to mine spatial correlations, resulting in a more compact representation for any input block x. The encoding transformation is shown in the following equation:

[0126]

[0127] in y represents the parameters of the transformed network, and y represents the latent features obtained from the compressed model.

[0128] The decoding process is the decoding module part, and the formula is expressed as:

[0129]

[0130] in Representing encoded features using g s The decoding process is performed to obtain the final reconstructed image x.

[0131] For background encoding, a simple multi-scale sparse convolution is used for implicit feature extraction. Since the background region contains less usable information, the encoding can be lossy to minimize the bit rate. For ROI region encoding, a larger PSNR should be ensured as much as possible.

[0132] In one embodiment of this application, such as Figure 2 As shown, before the decoding module 230 processes the compressed point cloud data, the method further includes:

[0133] S1: Input the compressed point cloud data into the super prior encoder 220, and perform upsampling and downsampling operations on the compressed point cloud data through the super prior encoder 220 to obtain the super prior point cloud data features.

[0134] First, the compressed point cloud data is downsampled to obtain the super-prior downsampled global features.

[0135] Specifically, the super-prior encoder includes a multilayer perceptron and a position-free global feature extraction module. The downsampling operation is shown below:

[0136] S101, the compressed point cloud data is input into the multilayer perceptron 215, and then the output of the multilayer perceptron 215 is input into the positionless global feature extraction module 222.

[0137] S102, when i=2, the output of the first positionless global feature extraction module (i.e., positionless global feature extraction module 222) is input into the second multilayer perceptron (i.e., multilayer perceptron 215) to obtain the second perceptual feature.

[0138] The second perceptual feature and spatial coordinates are input into the second positionless global feature extraction module (i.e., positionless global feature extraction module 222) for global feature extraction to obtain the second super-prior downsampled global feature.

[0139] S103, when i=3, the output of the second positionless global feature extraction module (i.e., the output of the second super-prior downsampled global feature of the positionless global feature extraction module 222) is input into the third multilayer perceptron (i.e., multilayer perceptron 215) to obtain the third perceptual feature.

[0140] The third perceptual feature and spatial coordinates are input into the third positionless global feature extraction module (i.e., positionless global feature extraction module 222) for global feature extraction to obtain the third super-prior downsampled global feature.

[0141] S104, after confirming that the prior compression bit rate of the point cloud data meets the requirements, the third super-prior downsampled global feature is used as the Gaussian distribution downsampled global feature.

[0142] For example, the prior compression bitrate requirement requires downsampling the compressed point cloud data by 32 times. It is understood that the size of the compressed point cloud data obtained by the multilayer perceptron 215 and the positionless global feature extraction module 222 in this embodiment is 1×256. The multilayer perceptron 215 and the positionless global feature extraction module 222 downsample the compressed point cloud data to 256 / 4, and then further downsample it to 256 / 32 to obtain the super-prior downsampled global features.

[0143] It is understandable that i represents the position of the positionless global feature extraction module and the multilayer perceptron in the model.

[0144] Therefore, this embodiment of the application can further compress the point cloud data by downsampling the compressed point cloud data. Through the combination of multilayer perceptron and positionless global feature extraction module, it can extract deep global features of the point cloud data, thereby improving the accuracy of the model in the subsequent recognition and reconstruction process.

[0145] After obtaining Gaussian distributed downsampled global features by downsampling the compressed point cloud data, it is necessary to perform arithmetic encoding on the hyperprior downsampled global features. The specific process is as follows:

[0146] Step 1: As Figure 2 As shown, the super-prior downsampled global features are input into the quantization module Q, and the super-prior downsampled global features are quantized by the quantization module Q.

[0147] Step 2: Input the quantized features into the arithmetic coding module AE to obtain arithmetic coding features.

[0148] In other words, the quantized features are transformed into a string of binary files using the arithmetic encoding module (AE).

[0149] Step 3: Input the arithmetic encoded features into the arithmetic decoding module AD, and perform arithmetic decoding operation on the arithmetic encoded features through the arithmetic decoding module AD to obtain the downsampled decoded global features.

[0150] Then, the global features of the super-prior downsampling are upsampled to obtain the super-prior point cloud data features.

[0151] In other words, the process of upsampling the global features of the super-prior downsampling to obtain the super-prior point cloud data features includes: upsampling the global features of the downsampled decoding to obtain the super-prior point cloud data features.

[0152] Specifically, firstly, the downsampled decoded global features and spatial coordinates of size 256 / 32 are input into the positionless global feature extraction module 222, and then the output of the positionless global feature extraction module 222 is input into the multilayer perceptron 215. Next, the output of the multilayer perceptron 215 is input into the positionless global feature extraction module 222, and then the output of the positionless global feature extraction module 222 is input into the multilayer perceptron 215, upsampling the downsampled decoded global features to a feature size of 256 / 4. Finally, the feature from the multilayer perceptron 215 is input into the positionless global feature extraction module 222, and then the output of the positionless global feature extraction module 222 is input into the multilayer perceptron 215, upsampling the downsampled decoded global features to a feature size of 1×256, thus obtaining the super-prior point cloud data features.

[0153] Therefore, by upsampling and downsampling the compressed point cloud data, this embodiment of the application can extract more representative global features, thereby obtaining the features of the entire region.

[0154] S2: Obtain Gaussian distribution parameters based on the features of the prior point cloud data.

[0155] In other words, such as Figure 2 As shown, after the super-prior encoder 220 obtains the super-prior point cloud data features, it inputs the super-prior point cloud data features into the Gaussian parameter calculation module CM, and obtains the Gaussian distribution parameters through the Gaussian parameter calculation module CM. The Gaussian distribution parameters include the mean and variance.

[0156] S3: Perform arithmetic encoding and decoding operations on the compressed point cloud data based on Gaussian distribution parameters to obtain the target compressed point cloud data.

[0157] In other words, such as Figure 3 As shown, before decoding the target compressed point cloud data, it is necessary to perform arithmetic encoding and decoding operations on the compressed point cloud data using Gaussian distribution parameters. The specific steps are as follows:

[0158] S301, the compressed point cloud data is input into the quantization module Q, and the compressed point cloud data is quantized by the quantization module Q to obtain the quantized compressed point cloud features.

[0159] S302, the quantized compressed point cloud features and the prior point cloud data features are input into the quantization module Q, and the Gaussian distribution parameters are calculated through the quantization module Q.

[0160] S303 inputs the quantized compressed point cloud features and Gaussian distribution parameters into the arithmetic coding module AE. The arithmetic coding module AE uses the Gaussian distribution parameters to perform arithmetic coding operations on the quantized compressed point cloud features.

[0161] S304. The quantized compressed point cloud features and Gaussian distribution parameters after arithmetic encoding are input into the arithmetic decoding module AD. The arithmetic decoding module AD uses the Gaussian distribution parameters to decode the quantized compressed point cloud features after arithmetic encoding to obtain the target compressed point cloud data.

[0162] Therefore, by performing upsampling, downsampling, and arithmetic encoding / decoding operations on the compressed point cloud data, this embodiment of the application can extract more representative global features, thereby obtaining the features of the entire region and improving the accuracy of the model in image processing of the target compressed point cloud data.

[0163] As a specific embodiment of this application, the compression model proposed in this application includes multiple Local-Global (LGB) blocks composed of global feature extraction modules and local feature extraction modules, which can better capture the global information of the point cloud data to be compressed. Furthermore, this application designs a multi-scale feature pyramid structure (i.e., each LGB acquires all the features preceding it) to fully utilize spatial correlation. Experiments demonstrate the effectiveness of the compression model in this embodiment. To better perform downsampling operations on the point cloud data to be compressed, four downsampling LGB blocks are stacked. The input of the compression model is represented as an N×3 matrix, with downsampling scales of 4x, 16x, and 16x.

[0164] The decoding module includes a decoding convolutional layer and a positionless global feature extraction module. The decoding convolutional layer contains deconvolution for upsampling, while the positionless global feature extraction module is a positionless embedding Transformer structure.

[0165] The super-prior encoder shares a similar network structure with the compressed model. The super-prior point cloud data features output by the super-prior decoder are used to calculate the Gaussian distribution parameters. The following equations (7)(8)(9) represent the computation process of the positionless global feature extraction module. Each positionless global feature extraction module consists of at least one multi-head self-attention module and one feedforward neural network (FFN), with residual connections around each module, followed by processing using a standard regularization module (i.e., a normalization module).

[0166] FFN(x)=max(0,xW1+b1)W2+b2 (7)

[0167] Layer Normal(X+sublayer(X)) (8)

[0168]

[0169] Where FFN represents a feedforward neural network, Attention TNPE The output features of the self-attention module are represented by Q, K, V, and softmax, where Q represents the query, K represents the key, V represents the value, and d represents the fully connected layer. K The dimension of the feature is represented by X, which represents the spatial coordinates and downsampled local features of the point cloud data. sublayer(X) represents the normalization of X, and W1, b1, W2 and b2 represent the weight parameter matrices.

[0170] Furthermore, the recognition module shares parameters with the compression model. During training, the compression model is first trained and the quantized bitstream is saved. Then, the decoding module is used to classify the categories based on the reconstructed features (Bx1x256). Finally, the recognition module uses Softmax for classification; for example, if the input is ModelNet40, the output is (Bx40).

[0171] S330 performs image processing based on compressed point cloud data.

[0172] In one embodiment of this application, S330 includes: classifying the target compressed point cloud data.

[0173] In other words, such as Figure 2 As shown, after obtaining the target compressed point cloud data, it can be directly input into the recognition module 240. As an example, the recognition module 240 consists of a four-layer multilayer perceptron 215, ultimately outputting the type of the point cloud data. For example, the type of the point cloud data is "aircraft".

[0174] In another embodiment of this application, S330 includes: classifying the target compressed point cloud data and reconstructing the target scene or target object based on the target compressed point cloud data.

[0175] In other words, the embodiments of this application, geared towards machine-perceptive point cloud classification tasks, can simultaneously complete point cloud reconstruction and point cloud classification tasks. For example... Figure 2 As shown, according to the reconstruction requirements, this embodiment of the application directly inputs the target compressed point cloud data into the recognition module 240 and uses the target compressed point cloud data for reconstruction, avoiding the use of all point cloud data for reconstruction, thereby reducing transmission costs, and thus improving classification accuracy without changing the compression model and decoding module structure.

[0176] In this embodiment, a positionless Transformer structure (i.e., a positionless global feature extraction module) is used in the decoding module. This structure can effectively utilize its own structural information to calculate attention during the upsampling and reconstruction of the image, thereby bringing performance gains to the overall model.

[0177] Specifically, such as Figure 2 As shown, when the model has both classification and reconstruction tasks, the target compressed point cloud data can be simultaneously input into the decoding module 230 and the recognition module 240.

[0178] In other words, such as Figure 2As shown, the target compressed point cloud data is input into the decoding module 230, and the target scene or target object is obtained through the decoding module 230. The decoding module 230 includes: a multi-layer point cloud decoding module, each of which further includes a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer.

[0179] When the target compressed point cloud data is input into the decoding module 230, the specific reconstruction steps are as follows:

[0180] Step 1: Input the target compressed point cloud data of size N / 16 into the decoding convolutional layer 235. The target compressed point cloud data is decoded by the decoding convolutional layer 235. Then, the output of the decoding convolutional layer 235 is input into the positionless global feature extraction module 222 to obtain the first decoded point cloud data.

[0181] Step 2: Input the first decoded point cloud data of size N / 16 into the decoding convolutional layer 235, decode the first decoded point cloud data through the decoding convolutional layer 235, and then input the output of the decoding convolutional layer 235 into the positionless global feature extraction module 222 to obtain the second decoded point cloud data.

[0182] Step 3: Input the second decoded point cloud data into the decoding convolutional layer 235, decode the second decoded point cloud data through the decoding convolutional layer 235, and then input the output of the decoding convolutional layer 235 into the positionless global feature extraction module 222 for upsampling to obtain the third decoded point cloud data of size N / 4.

[0183] Step four: Input the third decoded point cloud data into the multilayer perceptron 215. The multilayer perceptron 215 upsamples the third decoded point cloud data to finally obtain the reconstructed image, that is, the image corresponding to the reconstructed target scene or target object.

[0184] It is understood that in the current embodiment, the target compressed point cloud data is input into the decoding module 230 and the recognition module 240 respectively to perform recognition and reconstruction tasks. The specific implementation process in the recognition module is described in the previous embodiment and will not be repeated here.

[0185] Therefore, the embodiments of this application decode the target compressed point cloud data by decoding the convolutional layer and the positionless global feature extraction module, which can use the global features of the point cloud data to reconstruct the target scene or target object, thereby improving the accuracy of reconstruction.

[0186] In another embodiment of this application, S330 includes: reconstructing the target scene or target object based on the target compressed point cloud data.

[0187] It is understood that in the current embodiment, the target processing point cloud data model only inputs the target compressed point cloud data into the decoding module 230 for reconstruction operation to obtain the reconstructed image corresponding to the target scene or target object. The reconstruction process is described in the previous embodiment and will not be repeated here.

[0188] Therefore, by performing image processing on the target compressed point cloud data after compression, the embodiments of this application can improve the rate-distortion performance and classification accuracy of the model.

[0189] Therefore, this application proposes an end-to-end optimized encoding / decoding model to compress point cloud data, jointly optimizing the compression ratio, semantic classification accuracy, and signal reconstruction quality. The point cloud encoding / decoding uses a Transformer structure for local and global feature extraction, which can better capture short-term and long-term feature representations using point cloud location information, and better fuse local and global features, thus achieving excellent compression performance. The point cloud classification task is performed directly on the decoded features based on the compression, allowing for the selection of appropriate features for reconstruction according to the needs of the decoding end. Experiments show that the method in this application, compared to methods such as Geometry-based Compression (GPCC), achieves lower compression ratios and higher PSNR gains on datasets such as ShapeNetCoreV2 and ModelNet40, demonstrating good compression and classification performance at low bit rates.

[0190] Therefore, this application combines Transformer and multilayer perceptron as basic units to analyze and aggregate short-range and long-range information for point cloud reconstruction and classification, thereby improving rate-distortion performance and classification accuracy.

[0191] The foregoing described a target processing point cloud data model in an embodiment of this application; the following describes an apparatus for processing point cloud data.

[0192] like Figure 6 As shown, an apparatus 800 for processing point cloud data includes: a data acquisition module 810, a data compression module 820, and an image processing module 830.

[0193] The data acquisition module 810 is configured to acquire point cloud data to be compressed, wherein the point cloud data to be compressed is obtained by collecting point cloud data of a target scene or target object.

[0194] The data compression module 820 is configured to input the point cloud data to be compressed into a compression model and obtain compressed point cloud data through the compression model. The compression model includes a multi-layer feature extraction module, each of which further includes a local feature extraction module and a global feature extraction module connected to the local feature extraction module. The global feature extraction module includes at least a self-attention module and a feedforward neural network.

[0195] The image processing module 830 is configured to perform image processing based on the compressed point cloud data.

[0196] In one embodiment of this application, the global feature extraction module further includes: a first normalization module and a second normalization module; wherein, the first normalization module is configured to: input the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data into the first normalization module; perform a summation operation on the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data through the first normalization module to obtain a first summed feature; perform a normalization operation on the first summed feature to obtain a first normalized feature; the second normalization module is configured to: input the output of the feedforward neural network and the first normalized feature into the second normalization module; perform a summation operation on the output of the feedforward neural network and the first normalized feature through the second normalization module to obtain a second summed feature; perform a normalization operation on the second summed feature to obtain a downsampled global feature.

[0197] In one embodiment of this application, the data compression module 820 is configured to: input the output of the (i-1)th global feature extraction module into the i-th local feature extraction module for local feature extraction to obtain the i-th downsampled local feature; input the output of the (i-1)th global feature extraction module and the i-th downsampled local feature into the i-th global feature extraction module to obtain the i-th downsampled global feature; wherein i is an integer greater than or equal to 1, the i-th downsampled global feature is the data obtained after compressing the point cloud data at a target ratio, and the target ratio is determined by all downsampled convolution kernels in the i-th global feature extraction module; repeat the above steps until the compression bitrate of the point cloud data meets the requirements to obtain the target downsampled global feature; input the target downsampled global feature into a multilayer perceptron to obtain compressed point cloud data.

[0198] In one embodiment of this application, the data compression module 820 is configured to: input the compressed point cloud data into a priori encoder; perform upsampling and downsampling operations on the compressed point cloud data using the priori encoder to obtain priori point cloud data features; obtain Gaussian distribution parameters based on the priori point cloud data features, wherein the Gaussian distribution parameters include mean and variance; and perform arithmetic encoding and decoding operations on the compressed point cloud data based on the Gaussian distribution parameters to obtain target compressed point cloud data. The image processing module 830 is configured to perform image processing based on the target compressed point cloud data.

[0199] In one embodiment of this application, the data compression module 820 is configured to: perform a downsampling operation on the compressed point cloud data to obtain super-prior downsampled global features; and perform an upsampling operation on the super-prior downsampled global features to obtain super-prior point cloud data features.

[0200] In one embodiment of this application, the super-prior encoder includes a multilayer perceptron and a positionless global feature extraction module; the data compression module 820 is configured to: input the output of the (i-1)th positionless global feature extraction module into the i-th multilayer perceptron to obtain the i-th perceptron feature; input the i-th perceptron feature and the spatial coordinates into the i-th positionless global feature extraction module for global feature extraction to obtain the i-th super-prior downsampled global feature; repeat the above steps until the prior compression bit rate of the point cloud data meets the requirements to obtain the super-prior downsampled global feature.

[0201] In one embodiment of this application, the image processing module 830 is configured to: classify the target compressed point cloud data; and / or reconstruct the target scene or the target object based on the target compressed point cloud data.

[0202] In one embodiment of this application, the image processing module 830 is configured to: input the target compressed point cloud data into the decoding module, and obtain the target scene or target object through the decoding module. The decoding module includes: a multi-layer point cloud decoding module, each layer of the point cloud decoding module further including a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer.

[0203] In one embodiment of this application, the method is applied to a target processing point cloud data model, which includes a compression model, a super-prior encoder, a decoding module, and a recognition module. The target processing point cloud data model is trained using a target loss function. The target loss function is characterized by both reconstruction result loss and recognition result loss. The reconstruction result loss is determined by the compression model, the super-prior encoder, and the decoding module, and the recognition result loss is determined by the compression model, the super-prior encoder, and the recognition module.

[0204] In one embodiment of this application, the data acquisition module 810 is configured to: input the training data to be compressed into the point cloud data model to be trained for training, and obtain the target point cloud data model, wherein the training data to be compressed is obtained by adding noise data to the original point cloud data.

[0205] In the embodiments of this application, Figure 6 The module shown can achieve Figures 1 to 5 Each process in the method embodiment. Figure 6 The operations and / or functions of each module in the document are respectively designed to achieve... Figures 1 to 5 The corresponding processes in the method embodiments are described above. For details, please refer to the descriptions in the above method embodiments; to avoid repetition, detailed descriptions are omitted here.

[0206] like Figure 7 As shown, this application provides an electronic device 900, including: a processor 910, a memory 920, and a bus 930. The processor is connected to the memory via the bus. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the method as described in any one of the above embodiments. For details, please refer to the description in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0207] The bus is used to enable direct communication between these components. In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an On-Premises Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor.

[0208] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores computer-readable instructions, which, when executed by the processor, can perform the methods described in the above embodiments.

[0209] Understandable. Figure 7 The structure shown is for illustrative purposes only and may include structures larger than [other structures]. Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0210] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a server, it implements any of the methods described in all the above embodiments. For details, please refer to the descriptions in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0211] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0212] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing point cloud data, characterized in that, An application is provided for a point cloud data model for target processing. This model includes a compression model, a super-prior encoder, a decoding module, and a recognition module. The target processing point cloud data model is trained using a target loss function. The super-prior encoder obtains Gaussian distribution parameters from the compressed point cloud data. The decoding module upsamples and decodes the compressed point cloud data using the Gaussian distribution parameters to obtain a reconstructed image. The target loss function is characterized by both the reconstruction result loss value and the recognition result loss value. The reconstruction result loss value is determined by the compression model, the super-prior encoder, and the decoding module. The recognition result loss value is also determined by the compression model, the super-prior encoder, and the recognition module. The method includes: Acquire point cloud data to be compressed, wherein the point cloud data to be compressed is obtained by collecting point cloud data of a target scene or target object; The point cloud data to be compressed is input into the compression model, and the compressed point cloud data is obtained through the compression model. The compression model includes: a multi-layer feature extraction module, each feature extraction module further includes a local feature extraction module and a global feature extraction module connected to the local feature extraction module, and the global feature extraction module includes at least a self-attention module and a feedforward neural network. The compressed point cloud data is input into a super-prior encoder, which performs upsampling and downsampling operations on the compressed point cloud data to obtain super-prior point cloud data features. This includes: downsampling the compressed point cloud data, further compressing the point cloud data, obtaining super-prior downsampled global features through a combination of a multilayer perceptron and a positionless global feature extraction module, and upsampling the super-prior downsampled global features to obtain super-prior point cloud data features. The positionless global feature extraction module is based on the Transformer framework. Its inputs are the spatial coordinates of the point cloud data and the output features of the multilayer perceptron; the positionless global feature extraction module does not require input of local features. Based on the features of the prior point cloud data, Gaussian distribution parameters are obtained, wherein the Gaussian distribution parameters include mean and variance; Based on the Gaussian distribution parameters, perform arithmetic encoding and decoding operations on the compressed point cloud data to obtain the target compressed point cloud data; Image processing is performed based on the target compressed point cloud data.

2. The method according to claim 1, characterized in that, The global feature extraction module further includes: a first normalization module and a second normalization module; The first normalization module is configured as follows: The output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data are input into the first normalization module. The first normalization module performs a summation operation on the output of the self-attention module, the downsampled local features, and the spatial coordinates of the point cloud data to obtain the first summed feature. Normalize the first summed feature to obtain the first normalized feature; The second normalization module is configured as follows: The output of the feedforward neural network and the first normalized feature are input into the second normalization module; The second normalization module performs a summation operation on the output of the feedforward neural network and the first normalized feature to obtain the second summed feature; Normalize the second summed feature to obtain the downsampled global feature.

3. The method according to claim 2, characterized in that, The process of obtaining compressed point cloud data through the compression model includes: The output of the (i-1)th global feature extraction module is input into the ith local feature extraction module to extract local features and obtain the ith downsampled local features. The output of the (i-1)th global feature extraction module and the i-th downsampled local feature are input into the i-th global feature extraction module to obtain the i-th downsampled global feature. Where i is an integer greater than or equal to 2, the i-th downsampled global feature is the data obtained after the point cloud data is compressed at the target ratio, and the target ratio is determined by all downsampled convolution kernels in the i-th global feature extraction module; Repeat the above steps until the compression bitrate of the point cloud data meets the requirements to obtain the target downsampled global features; The target downsampled global features are input into a multilayer perceptron to obtain compressed point cloud data.

4. The method according to claim 1, characterized in that, The super prior encoder includes a multilayer perceptron and a positionless global feature extraction module; The step of downsampling the compressed point cloud data to obtain advanced prior downsampled global features includes: The output of the (i-1)th positionless global feature extraction module is input into the i-th multilayer perceptron to obtain the i-th perceptron feature. The i-th perceptual feature and spatial coordinates are input into the i-th positionless global feature extraction module to perform global feature extraction, thereby obtaining the i-th super-prior downsampled global feature; Repeat the above steps until the prior compression bitrate of the point cloud data meets the requirements, and obtain the super-prior downsampled global features.

5. The method according to claim 1, characterized in that, The image processing based on the target compressed point cloud data includes: Classify the target compressed point cloud data accordingly; and / or The target scene or the target object is reconstructed based on the target compressed point cloud data.

6. The method according to claim 5, characterized in that, The step of reconstructing the target scene or the target object based on the target compressed point cloud data includes: The target compressed point cloud data is input into the decoding module, and the target scene or target object is obtained through the decoding module. The decoding module includes a multi-layer point cloud decoding module, and each layer of point cloud decoding module further includes a decoding convolutional layer and a positionless global feature extraction module connected to the decoding convolutional layer.

7. A target processing point cloud data model, characterized in that, The target processing point cloud data model includes a compression model, a super-prior encoder, a decoding module, and a recognition module. The target processing point cloud data model is trained using a target loss function. The super-prior encoder is used to obtain Gaussian distribution parameters based on the compressed point cloud data. The decoding module is used to upsample and decode the compressed point cloud data using the Gaussian distribution parameters to obtain a reconstructed image. The target loss function is characterized by both the reconstruction result loss value and the recognition result loss value. The reconstruction result loss value is determined by the compression model, the super-prior encoder, and the decoding module. The recognition result loss value is also determined by the compression model, the super-prior encoder, and the recognition module. The model includes: A reconstruction module is used to compress and reconstruct the point cloud data to be compressed, obtaining a reconstructed image. The reconstruction module includes: A compression model is used to downsample and encode the point cloud data to obtain compressed point cloud data. The compressed point cloud data is then input into a super-prior encoder, which performs upsampling and downsampling operations on the compressed point cloud data to obtain super-prior point cloud data features. This includes: downsampling the compressed point cloud data, further compressing the point cloud data, obtaining super-prior downsampled global features through a combination of a multilayer perceptron and a positionless global feature extraction module, and upsampling the super-prior downsampled global features to obtain super-prior point cloud data features. The positionless global feature extraction module is based on the Transformer framework. Its inputs are the spatial coordinates of the point cloud data and the output features of the multilayer perceptron; the module does not require input of local features. A priori encoder is used to obtain Gaussian distribution parameters based on the compressed point cloud data and based on the features of the priori point cloud data, wherein the Gaussian distribution parameters include mean and variance. The decoding module is used to upsample and decode the compressed point cloud data using the Gaussian distribution parameters to obtain the reconstructed image, wherein the reconstructed image is a target scene image or a target object image, and to perform arithmetic encoding and decoding operations on the compressed point cloud data based on the Gaussian distribution parameters to obtain target compressed point cloud data. The identification module is used to classify the compressed point cloud data to obtain the type of the point cloud data, and to perform image processing based on the target compressed point cloud data.

Images