Point cloud data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a point cloud data processing method and device, equipment and a storage medium, and relates to the technical field of point cloud data processing. The method comprises the following steps: partitioning initial point cloud data to obtain foreground point cloud data and background point cloud data, setting a background scale reference value to be smaller than a foreground scale reference value, compressing the foreground point cloud data by using a foreground encoder to obtain foreground point cloud compressed data, compressing the background point cloud data by using a background encoder, and obtaining background point cloud compressed data; the foreground point cloud compression data and the background point cloud compression data are decoded to obtain background point cloud compression data, the foreground point cloud compression data and the background point cloud compression data are decoded and spliced to obtain fused point cloud data, and data reconstruction is performed on the fused point cloud data to obtain reconstructed point cloud data. And the background point cloud data is compressed by using a higher compression rate. Therefore, compared with the background point cloud data, the compressed foreground point cloud data has more bit streams. The key information of the target object in the foreground part is reserved, the data volume of the background part is reduced, and the data accuracy and processing efficiency are improved.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud data processing, and in particular to point cloud data processing methods, devices, equipment and storage media. Background Art

[0002] Point cloud compression is widely used in the field of 3D data processing and transmission. It reduces the redundancy of point cloud data through compression encoding, significantly reduces the demand for storage space and transmission bandwidth, and thus improves the efficiency of point cloud data processing. Although point cloud compression technology improves the processing efficiency of point cloud data, it is difficult for the point cloud compression process in related technologies to simultaneously ensure the accuracy and efficiency of detection results. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a point cloud data processing method, device, equipment and storage medium to improve the data accuracy and data processing efficiency of point cloud compression results.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a point cloud data processing method, comprising:

[0005] Partitioning the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data at least includes a target object;

[0006] Acquire a foreground initial scale of the foreground point cloud data, acquire a foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data;

[0007] Acquire a background initial scale of the background point cloud data, acquire a background scale reference value based on the background initial scale, compress the background point cloud data using a background encoder to obtain background point cloud compressed data, wherein the background scale reference value is smaller than the foreground scale reference value;

[0008] The foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain fused point cloud data, and the fused point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

[0009] In some embodiments, the foreground encoder includes at least one cascaded foreground encoding unit, and the foreground point cloud data is compressed by the foreground encoder to obtain foreground point cloud compressed data having a scale equal to the foreground scale reference value, including:

[0010] Using the foreground encoding unit to encode the foreground input data to obtain a foreground encoding result, and obtaining a foreground intermediate scale parameter of the foreground encoding result, wherein the foreground input data of the first foreground encoding unit is the foreground point cloud data;

[0011] If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is used as the foreground input data of the next foreground encoding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground encoding result corresponding to the foreground intermediate scale parameter is used as the foreground point cloud compression data.

[0012] In some embodiments, the foreground encoding unit is used to encode the foreground input data to obtain a foreground encoding result, including:

[0013] Downsampling the foreground input data to obtain first downsampled data;

[0014] Inputting the first down-sampled data into a point cloud data processing model for data processing to obtain first point cloud compressed data;

[0015] Downsampling the first point cloud compressed data to obtain second downsampled data;

[0016] Channel feature processing is performed on the second down-sampled data to obtain the foreground encoding result.

[0017] In some embodiments, the point cloud data processing model includes a downsampling extraction submodel and an upsampling extraction submodel, the downsampling extraction submodel includes a cascaded first number of downsampling extraction units, the upsampling extraction submodel includes a cascaded first number of upsampling extraction units, the downsampling extraction units and the upsampling extraction units at symmetrical positions constitute a sampling group, and the first downsampling data is input into the point cloud data processing model for data processing to obtain first point cloud compressed data, including:

[0018] Inputting the first down-sampling data into the down-sampling extraction sub-model for feature extraction, obtaining first output data of each down-sampling extraction unit, wherein the first input data of a subsequent down-sampling extraction unit is the first output data of a previous down-sampling extraction unit;

[0019] Input the first output data into the upsampling extraction sub-model for feature extraction, and obtain the second output data of each upsampling extraction unit, wherein the second input data of the upsampling extraction unit is the second output data of the previous upsampling extraction unit and the first output data of the downsampling extraction unit in the sampling group;

[0020] A convolution operation is performed on the second output data of the last upsampling extraction unit to obtain third output data, and points with null coordinate values ​​in the third output data are deleted to obtain the first point cloud compressed data.

[0021] In some embodiments, the downsampling extraction unit includes a sparse convolution layer and a feature extraction unit, the feature extraction unit includes a cascaded first convolution processing submodule, a local receptive field module and a second convolution processing submodule; the step of the downsampling extraction unit obtaining the first output data according to the first input data includes:

[0022] Using the sparse convolution layer to perform a convolution operation on the first input data to obtain sparse convolution processed data;

[0023] Using the first convolution processing submodule to perform a convolution operation on the sparse convolution processing data to obtain first intermediate processing data;

[0024] Using the local receptive field module to perform attention enhancement on the first intermediate processed data to obtain second intermediate processed data;

[0025] The second convolution processing submodule is used to perform a convolution operation on the second intermediate processed data to obtain the first output data.

[0026] In some embodiments, the first convolution processing submodule includes a first convolution group and a second convolution group, the first convolution group includes a plurality of first convolution layers in cascade, and the second convolution group includes a plurality of second convolution layers in cascade; and the using the first convolution processing submodule to perform a convolution operation on the sparse convolution processing data to obtain first intermediate processing data includes:

[0027] Inputting the sparse convolution processing data into the first convolution group to obtain the first intermediate convolution data of the last first convolution layer;

[0028] Inputting the sparse convolution processed data into the second convolution group to obtain the second intermediate convolution data of the last second convolution layer;

[0029] The first intermediate convolution data and the second intermediate convolution data are concatenated to obtain the first intermediate processed data.

[0030] In some embodiments, the local receptive field module includes a second number of attention units, a local receptive field convolution layer, and a local receptive field output layer; and using the local receptive field module to perform attention enhancement on the first intermediate processed data to obtain second intermediate processed data includes:

[0031] Inputting the first intermediate processed data into the attention unit for attention enhancement to obtain first local data;

[0032] Accumulating the first local data to obtain second local data, and inputting the second local data into the local receptive field convolution layer for convolution operation to obtain third local data;

[0033] The third local data is input into the local receptive field output layer for feature extraction to obtain the second intermediate processed data.

[0034] In some embodiments, the attention unit includes a first attention convolution layer, a channel attention module, and a second attention convolution layer; the first intermediate processed data is respectively input into the attention unit for attention enhancement to obtain the first local data, including:

[0035] Inputting the first intermediate processed data into the first attention convolution layer for convolution operation to obtain first attention intermediate data;

[0036] Inputting the first attention intermediate data into the channel attention module for attention calculation to obtain second attention intermediate data;

[0037] The second attention intermediate data is input into the second attention convolution layer for convolution operation to obtain the first local data.

[0038] In some embodiments, inputting the third local data into the local receptive field output layer for feature extraction to obtain the second intermediate processed data includes:

[0039] Performing a softmax calculation on the third local data to obtain an output weight corresponding to the second quantity;

[0040] The output weight is used to perform weighted calculation on the second attention intermediate data to obtain the second intermediate processed data.

[0041] In some embodiments, performing channel feature processing on the second down-sampled data to obtain the foreground encoding result includes:

[0042] Performing feature fusion and then upsampling on the second downsampled data to obtain channel feature data, wherein the channel feature data includes a preset number of extended voxels corresponding to each basic voxel in the second downsampled data;

[0043] The extended voxels of the same basic voxel in the channel feature data are obtained to form a voxel group, and probability estimation is performed on the extended voxels in the voxel group to obtain the foreground encoding result.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a point cloud data processing device, comprising:

[0045] Point cloud segmentation module: used to partition the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data at least includes the target object;

[0046] A foreground compression module is used to obtain a foreground initial scale of the foreground point cloud data, obtain a foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data;

[0047] Background compression module: used to obtain the background initial scale of the background point cloud data, obtain the background scale reference value based on the background initial scale, compress the background point cloud data using a background encoder to obtain background point cloud compressed data, and the background scale reference value is smaller than the foreground scale reference value;

[0048] Point cloud reconstruction module: used to decode the foreground point cloud compressed data and the background point cloud compressed data respectively to obtain foreground point cloud decoded data and background point cloud decoded data, splice the foreground point cloud decoded data and the background point cloud decoded data to obtain fused point cloud data, input the fused point cloud data into the point cloud reconstruction model for data reconstruction, and obtain reconstructed point cloud data.

[0049] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0051] The point cloud data processing method, device, equipment and storage medium proposed in the embodiment of the present application obtain foreground point cloud data and background point cloud data by partitioning the initial point cloud data, wherein the foreground point cloud data at least includes the target object. Next, the foreground initial scale of the foreground point cloud data is obtained, the foreground scale reference value is obtained based on the foreground initial scale, and the foreground point cloud data is compressed using the foreground encoder to obtain foreground point cloud compressed data. And the background initial scale of the background point cloud data is obtained, the background scale reference value is obtained based on the background initial scale, and the background point cloud data is compressed using the background encoder to obtain background point cloud compressed data, wherein the background scale reference value is less than the foreground scale reference value. Finally, the foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain fused point cloud data, and the fused point cloud data is input into the point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In the embodiment of the present application, the initial point cloud data is first partitioned to obtain foreground point cloud data containing the target object and background point cloud data not containing the target object. Next, the background scale reference value is set smaller than the foreground scale reference value to compress the background point cloud data with a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the amount of data in the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the point cloud data processing method provided in an embodiment of the present application.

[0053] Figure 2 It is a schematic diagram of obtaining foreground point cloud data and background point cloud data by partitioning provided in an embodiment of the present application.

[0054] Figure 3 It is a schematic diagram of compressing foreground point cloud data and background point cloud data provided by an embodiment of the present application.

[0055] Figure 4 This is a flowchart of using a foreground encoding unit to encode foreground input data to obtain a foreground encoding result provided by an embodiment of the present application.

[0056] Figure 5 It is a schematic diagram of the coding principle of the foreground coding unit and the Beijing coding unit provided in the embodiment of the present application.

[0057] Figure 6 It is a schematic diagram of the model structure of the point cloud data processing model provided in an embodiment of the present application.

[0058] Figure 7 It is a flowchart provided by an embodiment of the present application for inputting the first down-sampled data into a point cloud data processing model for data processing to obtain first point cloud compressed data.

[0059] Figure 8 It is a structural diagram of a downsampling extraction unit provided in an embodiment of the present invention.

[0060] Fig. 9 It is a flowchart of a downsampling extraction unit provided in an embodiment of the present application obtaining first output data based on first input data.

[0061] Fig.10 It is a structural diagram of the first convolution processing submodule provided in an embodiment of the present application.

[0062] Fig.11 It is a structural diagram of the local receptive field module provided in an embodiment of the present application.

[0063] Fig.12 This is a schematic diagram of the impact of different scale factors on the detection task and compression task in the test scenario of the embodiment of the present application.

[0064] Fig.13 It is a schematic diagram of compression performance in the test scenario provided in an embodiment of the present application.

[0065] Fig.14 This is a structural block diagram of a point cloud data processing device provided in another embodiment of the present application.

[0066] Fig.15 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] First, some nouns involved in this application are analyzed:

[0071] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] Point cloud compression is widely used in the field of 3D data processing and transmission. It reduces the redundancy of point cloud data through compression encoding, significantly reduces the demand for storage space and transmission bandwidth, and thus improves the efficiency of point cloud data processing. In high-precision application scenarios such as autonomous driving and virtual reality, point cloud data is often used to build accurate 3D models to support complex visual inspection tasks.

[0073] Although point cloud compression technology improves the processing efficiency of point cloud data, if a higher compression rate is selected in the point cloud compression method in the related technology, more geometric information will be lost. On the contrary, if a lower compression rate is selected, the amount of point cloud data will be huge, affecting the data processing efficiency. Therefore, it is difficult to ensure the accuracy and efficiency of the detection results at the same time.

[0074] Based on this, the embodiments of the present application provide a point cloud data processing method, apparatus, device and storage medium. First, the initial point cloud data is partitioned to obtain foreground point cloud data containing the target object and background point cloud data that does not contain the target object. Next, the background scale reference value is set to be smaller than the foreground scale reference value to compress the background point cloud data with a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the amount of data in the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression results.

[0075] The embodiments of the present application provide a point cloud data processing method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the point cloud data processing method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0077] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0078] The point cloud data processing method provided in the embodiment of the present application relates to the technical field of point cloud data processing. The point cloud data processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be a computer program running in a terminal or a server. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports point cloud data processing; it can also be a small program, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The point cloud data processing method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0079] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the point cloud data processing method of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a point-to-point (P2P, Peer To Peer, P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (USB) or network communication connection methods, which are not limited in this embodiment.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] The point cloud data processing method in the embodiment of the present application is described below.

[0083] Figure 1 is an optional flow chart of the point cloud data processing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0084] Step 110: partition the initial point cloud data to obtain foreground point cloud data and background point cloud data.

[0085] In one embodiment, the initial point cloud data includes the target object and the related background. Therefore, the embodiment of the present application first partitions the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data includes the target object, and the background point cloud data does not include the target object. Therefore, the foreground point cloud data is considered to be the region of interest. For the region of interest, its corresponding geometric information should be retained as much as possible during the point cloud compression process to improve the task accuracy in visual tasks such as detection and recognition. The data information in the background point cloud data is more redundant, and the compression rate can be increased to reduce the amount of data to be processed, thereby improving data processing efficiency. Generally speaking, the amount of data in the foreground point cloud data is much smaller than that of the background point cloud data.

[0086] In one embodiment, referring to Figure 2 , Figure 2: This is a schematic diagram of partitioning to obtain foreground point cloud data and background point cloud data provided by an embodiment of the present application. First, the initial point cloud data is quantized and the floating point coordinates are converted into positive integer coordinates. Then, the quantized data is input into the point cloud segmentation network for segmentation. Once the segmentation is completed, all point clouds containing the target object are classified as foreground point cloud data, and point clouds that do not contain the target object are classified as background point cloud data. The point cloud segmentation network here can be a spatio-temporal convolutional network (spatio-temporal ConvNets).

[0087] Next, the respective encoding processes of the foreground point cloud data and the background point cloud data are described.

[0088] Step 120: obtaining a foreground initial scale of the foreground point cloud data, obtaining a foreground scale reference value based on the foreground initial scale, and compressing the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data.

[0089] In one embodiment, different point cloud data have different geometric accuracy. Therefore, the embodiment of the present application first obtains scale data characterizing the geometric accuracy. For the foreground point cloud data, for example, the specific foreground initial scale can be obtained according to the density or sampling interval of the point cloud of the foreground point cloud data.

[0090] Next, a foreground scale reference value is obtained based on the foreground initial scale, wherein the foreground scale reference value is used to limit the compression rate of the point cloud compression, that is, the scale of the compressed foreground point cloud data after compression should be close to the foreground scale reference value. Since the embodiment of the present application needs to retain the geometric features of the foreground part, the foreground scale reference value can be selected to be close to the foreground initial scale. For example, if the foreground initial scale is N1, N1-2 can be selected as the foreground scale reference value M1.

[0091] In one embodiment, after determining the foreground scale reference value, the point cloud compression process begins. Figure 3 , Figure 3 It is a schematic diagram of compressing foreground point cloud data and background point cloud data provided by an embodiment of the present application.

[0092] Since the foreground encoder includes at least one cascaded foreground coding unit, Figure 3The compression process of the foreground point cloud data is specifically described as follows: the foreground input data is encoded by the foreground coding unit to obtain a foreground coding result, and the foreground intermediate scale parameter of the foreground coding result is obtained. The foreground input data of the first foreground coding unit is the foreground point cloud data. If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is used as the foreground input data of the next foreground coding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground coding result corresponding to the foreground intermediate scale parameter is used as the foreground point cloud compression data.

[0093] That is to say, the foreground encoder includes many cascaded foreground coding units. In the actual application process, since the foreground initial scale of the specific foreground point cloud data must be determined according to the actual situation, it is not certain which foreground coding unit outputs the foreground coding result that can be used as the final foreground point cloud compression data. Therefore, the data is processed step by step, and then the foreground intermediate scale parameter of the foreground coding result of each foreground coding unit is judged. If the foreground intermediate scale parameter meets the requirements of the foreground scale reference value, considering the numerical deviation of the actual process, meeting the requirements does not mean that the foreground intermediate scale parameter must be equal to the foreground scale reference value. When the foreground intermediate scale parameter is slightly greater than or equal to the foreground scale reference value, it can be considered that the foreground intermediate scale parameter matches the foreground scale reference value. At this time, the foreground coding result corresponding to the foreground intermediate scale parameter is output as the foreground point cloud compression data output.

[0094] In one embodiment, referring to Figure 4 , Figure 4 The flowchart of the embodiment of the present application is to use the foreground encoding unit to encode the foreground input data to obtain the foreground encoding result, which specifically includes the following steps:

[0095] Step 410: down-sample the foreground input data to obtain first down-sampled data.

[0096] In one embodiment, referring to Figure 5 , Figure 5 This is a schematic diagram of the coding principle of the foreground coding unit and the Beijing coding unit provided in the embodiment of the present application. It can be seen that the foreground coding unit sequentially performs the first downsampling, data processing, second downsampling, and channel feature processing steps on the foreground input data to obtain the foreground coding result until the foreground intermediate scale parameter matches the foreground scale reference value. Among them, the foreground input data is first downsampled to obtain the first downsampled data.

[0097] Step 420: Input the first down-sampled data into the point cloud data processing model for data processing to obtain first point cloud compressed data.

[0098] In one embodiment, in combination Figure 5 Next, the first down-sampled data is input into the point cloud data processing model for data processing to obtain the first point cloud compressed data. This process is a lossy compression process.

[0099] Reference Figure 6 , Figure 6 It is a schematic diagram of the model structure of the point cloud data processing model provided in an embodiment of the present application. Figure 6 The midpoint cloud data processing model includes: a downsampling extraction submodel and an upsampling extraction submodel, wherein the downsampling extraction submodel includes a cascade of a first number of downsampling extraction units, and the first number is 5 for illustration in the figure, which does not mean that the value of the first number can only be 5. Similarly, the upsampling extraction submodel includes a cascade of a first number of upsampling extraction units, and the downsampling extraction units and the upsampling extraction units in symmetrical positions constitute a sampling group. The symmetrical position here refers to axial symmetry, for example, the order of the downsampling extraction units is 1, 2, 3, 4, 5, and the order of the upsampling extraction units is 1, 2, 3, 4, 5. At this time, the first downsampling extraction unit and the fifth upsampling extraction unit are symmetrical positions, and the two constitute a sampling group, the second downsampling extraction unit and the fourth upsampling extraction unit are symmetrical positions, and the two constitute a sampling group, ..., the fifth downsampling extraction unit and the first upsampling extraction unit are symmetrical positions, and the two constitute a sampling group.

[0100] In one embodiment, based on Figure 6 The structure of the point cloud data processing model is shown in Figure 2. Figure 7 , Figure 7 The flowchart of inputting the first down-sampled data into the point cloud data processing model for data processing to obtain the first point cloud compressed data provided by the embodiment of the present application specifically includes the following steps:

[0101] Step 710: Input the first down-sampling data into the down-sampling extraction sub-model for feature extraction, and obtain the first output data of each down-sampling extraction unit.

[0102] In one embodiment, the first down-sampled data is input into the down-sampled extraction sub-model for feature extraction, referring to Figure 6 , and the data is processed by five down-sampling extraction units in sequence. Assuming that the input data of each down-sampling extraction unit is called the first input data, and the output data is called the first output data, at this time, the first input data of the first down-sampling extraction unit is the first down-sampling data, and the first input data of the next down-sampling extraction unit is the first output data of the previous down-sampling extraction unit. At the same time, the first output data of each down-sampling extraction unit needs to be obtained.

[0103] In one embodiment, referring to Figure 8 , Figure 8It is a structural diagram of a downsampling extraction unit provided in an embodiment of the present invention. Figure 8 The mid- and down-sampling extraction units include: a sparse convolution layer (S shown in the figure) and a feature extraction unit (FE shown in the figure), wherein the feature extraction unit includes: a cascaded first convolution processing submodule, a local receptive field module and a second convolution processing submodule.

[0104] In one embodiment, Figure 8 Based on the structure of the downsampling extraction unit, refer to Fig. 9 , Fig. 9 The downsampling extraction unit according to the embodiment of the present application obtains the first output data according to the first input data, including the following steps:

[0105] Step 910: Use a sparse convolution layer to perform a convolution operation on the first input data to obtain sparse convolution processed data.

[0106] In one embodiment, the sparse convolution layer is specifically used to process sparse data. Sparse data refers to data in which most elements are zero or close to zero. Unlike the convolution layer, which calculates all input elements, the sparse convolution layer only needs to calculate non-zero elements, which can improve computing efficiency. The specific process is: convert the first input data into a sparse representation, and only retain the non-zero elements and their position information. Next, according to the sparse representation, only the non-zero elements are convolved, and finally the convolution results are recombined into complete output data to obtain sparse convolution processing data.

[0107] Step 920: Use the first convolution processing submodule to perform a convolution operation on the sparse convolution processing data to obtain first intermediate processing data.

[0108] In one embodiment, referring to Fig.10 , Fig.10 It is a structural diagram of the first convolution processing submodule provided in an embodiment of the present application. Fig.10 The first convolution processing submodule includes a first convolution group and a second convolution group, wherein the first convolution group includes a plurality of first convolution layers in cascade. In the figure, three first convolution layers are taken as an example, and the convolution kernel sizes of the illustrated first convolution layers are respectively: 1×1×1, 3×3×3, and 1×1×1. The second convolution group includes a plurality of second convolution layers in cascade. In the figure, two second convolution layers are taken as an example, and the convolution kernel sizes of the illustrated second convolution layers are respectively: 3×3×3 and 3×3×3. It can be understood that the number and convolution kernel size of the first convolution layer and the second convolution layer can be set according to actual needs, and this embodiment is only for illustration.

[0109] In one embodiment, Fig.10Based on the structure of the present invention, the first convolution processing submodule is used to perform a convolution operation on the sparse convolution processing data to obtain the process of first intermediate processing data. The specific description is as follows: the sparse convolution processing data is input into the first convolution group to obtain the first intermediate convolution data of the last first convolution layer; at the same time, the sparse convolution processing data is input into the second convolution group to obtain the second intermediate convolution data of the last second convolution layer; and then the first intermediate convolution data and the second intermediate convolution data are concatenated to obtain the first intermediate processing data.

[0110] Step 930: Use the local receptive field module to perform attention enhancement on the first intermediate processed data to obtain second intermediate processed data.

[0111] In one embodiment, referring to Fig.11 , Fig.11 It is a structural diagram of the local receptive field module provided in an embodiment of the present application. Fig.11 The local receptive field module includes a second number of attention units, a local receptive field convolution layer, and a local receptive field output layer, wherein the second number is described by taking 4 as an example and does not represent a limitation thereto. And Fig.11 The attention unit includes: the first attention convolution layer, the channel attention module and the second attention convolution layer.

[0112] Therefore, combined with Fig.11 The structure of the local receptive field module of the first intermediate processing data is enhanced by using the local receptive field module to obtain the second intermediate processing data. The process is described as follows:

[0113] First, the first intermediate processed data are input into the attention unit for attention enhancement to obtain the first local data. That is, the same first intermediate processed data is input into the attention unit for parallel calculation. Fig.11 In the processing process of the attention unit, the first intermediate processed data is input into the first attention convolution layer for convolution operation to obtain the first attention intermediate data, and then the first attention intermediate data is input into the channel attention module for attention calculation to obtain the second attention intermediate data, and finally the second attention intermediate data is input into the second attention convolution layer for convolution operation to obtain the first local data.

[0114] In one embodiment, according to the order of the attention units, the convolution kernel sizes of the first attention convolution layer are: 3×3×3, 5×5×5, 7×7×7, 9×9×9. In addition, the channel attention module is Efficient Channel Attention (ECA), which generates channel attention through fast one-dimensional convolution, and its convolution kernel size can be adaptively determined by the nonlinear mapping of the channel dimension. Finally, the convolution kernel size of the second attention convolution layer of different attention units is the same, which is 1×1×1. It can be seen that different convolution kernel sizes are used in the attention unit to improve the feature extraction performance.

[0115] Next, after each attention unit obtains the corresponding first local data, the first local data is accumulated to obtain the second local data. After obtaining the second local data, it is input into the local receptive field convolution layer for convolution operation to obtain the third local data, wherein the local receptive field convolution layer is a convolution layer with a convolution kernel size of 1×1×1.

[0116] In one embodiment, the third local data is input into the local receptive field output layer for feature extraction to obtain the second intermediate processed data. In the local receptive field output layer, the third local data is subjected to softmax calculation to obtain an output weight corresponding to the second quantity, wherein there is a corresponding relationship between the output weight and the second attention intermediate data, and finally the second attention intermediate data is subjected to weighted calculation using the output weight to obtain the second intermediate processed data.

[0117] Reference Fig.11 , assuming that the first attention intermediate data in the four attention units are represented as: x1, x2, x3 and x4, the corresponding second attention intermediate data are represented as: x1', x2', x3' and x4', and the first local data are represented as: w1, w2, w3 and w4. After accumulation, the third local data obtained by inputting the local receptive field convolution layer for convolution operation is represented as w. The second intermediate processed data obtained by feature extraction of the local receptive field output layer are represented as f1, f2, f3 and f4, among which f1 corresponds to x1', f2 corresponds to x2', f3 corresponds to x3', f4 corresponds to x4', and the second intermediate processed data obtained after weighting is represented as x.

[0118] Step 940: Use the second convolution processing submodule to perform a convolution operation on the second intermediate processed data to obtain first output data.

[0119] In one embodiment, the second convolution processing submodule has the same structure as the first convolution processing submodule, and the corresponding convolution kernel size can be set according to actual needs, or the same model structure as the first convolution processing submodule can be used. After obtaining the second intermediate processing data in the embodiment of the present application, the second convolution processing submodule is used to perform a convolution operation on the second intermediate processing data to obtain the first output data.

[0120] Step 720: Input the first output data into the upsampling extraction sub-model for feature extraction, and obtain the second output data of each upsampling extraction unit.

[0121] In one embodiment, referring to Figure 6 The upsampling extraction sub-model includes 5 upsampling extraction units that form a sampling group with the downsampling extraction unit. Since these upsampling extraction units are in a cascade relationship, it is assumed that the input data of each upsampling extraction unit is called the second input data, and the output data is called the second output data. At this time, the second input data of the subsequent upsampling extraction unit is the second output data of the previous upsampling extraction unit and the first output data of the downsampling extraction unit in the sampling group.

[0122] For example, for the first up-sampling extraction unit, it forms a sampling group with the fifth down-sampling extraction unit. At this time, theoretically, the second input data of the first up-sampling extraction unit includes the first output data of the fifth down-sampling extraction unit and the second output data of the previous up-sampling extraction unit. Since the first up-sampling extraction unit is the first module of the down-sampling extraction sub-model, its second input data only includes the first output data of the fifth down-sampling extraction unit. For the fifth up-sampling extraction unit, it forms a sampling group with the first down-sampling extraction unit. At this time, the second input data of the fifth up-sampling extraction unit includes the first output data of the first down-sampling extraction unit and the second output data of the fourth up-sampling extraction unit. By analogy, the second input data of each down-sampling extraction unit can be determined, and its corresponding second output data can be calculated.

[0123] In one embodiment, referring to Figure 6 The downsampling extraction unit is similar to the upsampling extraction unit, including a deconvolution layer and a feature extraction unit (FE shown in the figure). The deconvolution layer includes two different types of deconvolution layers. For example, the first type is the MinkowskiConvolutionTranspose deconvolution layer, that is, Figure 6 The TS shown in the figure, the second type is the MinkowskiGenerativeConvolutionTranspose deconvolution layer, that is, Figure 6In the GTS shown in Figure 1, these two deconvolution layers are modules in the MinkowskiEngine library, which are used to perform different types of deconvolution operations. Figure 6 ,In order, the combination types of upsampling extraction units are TS+FE, TS+FE, TS+FE, GTS+FE, TS+FE.

[0124] Step 730: performing a convolution operation on the second output data of the last upsampling extraction unit to obtain third output data, deleting the point cloud with empty coordinate values ​​in the third output data, and obtaining first point cloud compressed data.

[0125] In one embodiment, referring to Figure 6 , performing a convolution operation on the second output data of the last upsampling extraction unit to obtain third output data, where the convolution operation may be a sparse convolution operation. Next, deleting points whose coordinate values ​​in the third output data are null values, the first point cloud compressed data can be obtained.

[0126] Step 430: down-sample the first point cloud compressed data to obtain second down-sampled data.

[0127] In one embodiment, in combination Figure 5 After obtaining the first point cloud compressed data, a second downsampling operation is performed on it to obtain the corresponding second downsampling data, in order to perform further compression.

[0128] Step 440: Perform channel feature processing on the second down-sampled data to obtain a foreground encoding result.

[0129] In one embodiment, channel feature processing is performed on the second downsampled data to learn the spatial correlation of the point cloud and perform probability estimation of voxels. The process of performing channel feature processing on the second downsampled data to obtain the foreground encoding result is described as follows:

[0130] First, the second down-sampled data is subjected to feature fusion and then up-sampled to obtain channel feature data. Among them, the input second down-sampled data is subjected to multi-layer processing and feature fusion through deep feature aggregation. First, the initial features are extracted from the second down-sampled data. These initial features may include basic information such as spatial position and local geometric shape. The extracted initial features are preliminarily fused through connection operations or addition operations. Next, the preliminarily fused features are subjected to multi-layer processing. In each layer of processing, features are further extracted through convolution operations and nonlinear activation functions. After multi-layer processing, features at different levels are integrated together to form the final channel fusion data. Next, the channel fusion data is up-sampled, and a convolution operation with a convolution kernel size of 2×2×2 is implemented to expand each voxel in the channel fusion data to a preset number of smaller voxels, such as 8, so as to increase the size and resolution of the feature map and obtain channel feature data. At this time, it is assumed that the voxels in the channel feature data are called extended voxels, and the voxels in the second down-sampled data are called basic voxels. After the above process, each basic voxel becomes corresponding 8 smaller extended voxels. These extended voxels observe the basic voxels from different perspectives. Each perspective corresponds to a specific feature representation, which can characterize the multi-dimensional information of the point cloud data.

[0131] Next, the extended voxels of the same basic voxel in the channel feature data are obtained to form a voxel group. Assuming that the number of basic voxels is n, there are n voxel groups at this time, and each voxel group contains 8 extended voxels. Probability estimation is performed on the extended voxels in the voxel group to obtain a foreground encoding result. Among them, different voxel groups can be probability estimated in parallel. Specifically, for the extended voxels in each voxel group, multiple stages of probability estimation are performed in sequence to calculate the occupancy probability of the extended voxels, for example, the number of stages is 8. In each stage, the probability that each extended voxel belongs to a specific category is calculated based on the channel feature data, and the specific category here is set according to the actual situation. In each stage, the occupancy probability of the extended voxel is updated based on the result calculated in the previous stage, and then the channel feature data is updated based on the occupancy probability. After multiple iterative processes, the final channel feature data is obtained as the foreground encoding result.

[0132] Step 130: Obtain a background initial scale of the background point cloud data, obtain a background scale reference value based on the background initial scale, compress the background point cloud data using a background encoder to obtain background point cloud compressed data, and the background scale reference value is smaller than the foreground scale reference value.

[0133] In one embodiment, the background initial scale of the background point cloud data is obtained in the same manner as the foreground initial scale, and then a background scale reference value is obtained based on the background initial scale, wherein the background scale reference value is used to limit the compression rate of the point cloud compression, that is, the scale of the compressed background point cloud data after compression should be close to the background scale reference value. Since the embodiment of the present application needs to reduce the amount of data in the background part, the foreground scale reference value can be much smaller than the value of the background initial scale. For example, if the background initial scale is N2, N2-9 can be selected as the background scale reference value M2.

[0134] Next, refer to Figure 3 , the background encoder includes at least one cascaded background encoding unit. The compression process of the background point cloud data is specifically described as: using the background encoding unit to encode the background input data to obtain the background encoding result, and obtaining the background intermediate scale parameter of the background encoding result, the background input data of the first background encoding unit is the background point cloud data. If the background intermediate scale parameter does not match the background scale reference value, the background intermediate scale parameter is used as the background input data of the next background encoding unit until the background intermediate scale parameter matches the background scale reference value, and the background encoding result corresponding to the background intermediate scale parameter is used as the background point cloud compression data.

[0135] In one embodiment, the structure of the background coding unit is similar to that of the foreground coding unit, and the difference between the two is mainly that the background scale reference value is different from the foreground scale reference value. Figure 5 In the background coding unit, the background input data is also subjected to the first downsampling, data processing, the second downsampling, and channel feature processing steps in sequence to obtain the background coding result until the background intermediate scale parameter matches the background scale reference value.

[0136] From the above, it can be seen that by referring to the processing process of the foreground point cloud compression data, the background point cloud compression data can be obtained.

[0137] In one embodiment, the number of bits allocated in the foreground point cloud data compression process and the background point cloud data compression process is different. Assuming that the total number of bits is R all , the bit stream of the foreground part is R foreground , the bit stream of the background part is R background , where R all =R foreground +R background The entire compression process is dedicated to allocating more bits to the foreground and fewer bits to the background.

[0138] Step 140: Decode the foreground point cloud compressed data and the background point cloud compressed data respectively to obtain foreground point cloud decoded data and background point cloud decoded data, splice the foreground point cloud decoded data and the background point cloud decoded data to obtain fused point cloud data, input the fused point cloud data into a point cloud reconstruction model for data reconstruction, and obtain reconstructed point cloud data.

[0139] In one embodiment, in combination Figure 3 , respectively decode the foreground point cloud compressed data and the background point cloud compressed data to obtain the foreground point cloud decoded data and the background point cloud decoded data. In this embodiment, the decoding process corresponds to the encoding process. Then, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain the fused point cloud data. At this time, the fused point cloud data contains the foreground part and the background part. Finally, the fused point cloud data is processed based on the point cloud reconstruction model to obtain the final reconstructed point cloud data. In this embodiment, the point cloud reconstruction model is Figure 6 The model structure of the point cloud data processing model shown in is consistent, and the model parameters vary according to the actual training process.

[0140] In one embodiment, after obtaining the reconstructed point cloud data, a downstream visual inspection task may be performed, and the inspection task is set according to actual needs.

[0141] The following describes the testing process of the point cloud data processing method provided in the embodiment of the present application.

[0142] First, the KITTI dataset is used to pre-train the relevant models used in the point cloud data processing method. The KITTI dataset is quantized to an accuracy of 2cm (14 bits). The dataset is then divided into a training set and a test set. In order to evaluate the compression performance, the number of bits per point (bpp) is used as an indicator of the compression ratio, and the point-to-point error (D1) and the point-to-plane error (D2) are used as distortion indicators. In the detection task, mAP@0.25 and mAP@0.5 are used as evaluation criteria. In addition, in order to make a fair experimental comparison, some models in the related technologies are selected for comparison, such as G-PCC TMC13-v14, AVS PCRM-v11.0, and PCGCv2. At the same time, the point cloud data processing method in the embodiment of this application is recorded as SparsePCGC. For downstream detection tasks, Group-Free is used as the detection network, the computer configuration is NVIDIA RTX A5000, and the code development is based on the PyTorch framework.

[0143] In one embodiment, the scale factor of the foreground part is recorded as Scalef, and the scale factor of the background part is recorded as Scaleb, which is used to reflect the different bit stream allocations for the foreground and background. Bppf / Bppb represents the bit ratio between the foreground and background areas, and Bpp represents the total bit stream. Fig.12, Fig.12 This is a schematic diagram of the impact of different scale factors on detection tasks and compression tasks in the test scenario of the embodiment of this application. Fig.12 From the data, we can see that the compression and detection task performance are different under different bit conditions. For example, the highlighted data under mAP@0.25 (Scale f=2) clearly shows that allocating more bits to the foreground area can maintain a high level of performance in the downstream detection task, clarifying the key importance of the foreground area in the detection task. However, when reducing the number of bits in the foreground area (such as Scale f=4), the detection task performance drops significantly, with an overall value of about 0.05.

[0144] Next, refer to Fig.13 , Fig.13 Schematic diagram of compression performance in the test scenario provided by the embodiment of the present application. Fig.13 The horizontal axis is Bpp, and the vertical axis is the corresponding values ​​of three different evaluation indicators: point-to-point error (D1), point-to-plane error (D2) and mAP@0.25. It shows the combination of reconstructed point clouds obtained by encoding the foreground and background parts respectively, which is used to evaluate its compression performance. Specifically, SAVD-ALL represents the compression performance when the foreground and background data are not encoded separately; and SAVD-SPLIT represents the performance of the method of the embodiment of the present application after encoding the foreground and background data separately. Obviously, SAVD-SPLIT is better than SAVD-ALL in terms of compression performance. It can be seen that the method of the embodiment of the present application is better than AVS PCRM and PCGCv2 in compression performance, and is comparable to G-PCC and SparsePCGC. It can be seen that under the same bit rate conditions, the method of the embodiment of the present application significantly improves the detection performance in the high bit rate range while maintaining good detection quality.

[0145] The point cloud data processing method provided in the embodiment of the present application obtains foreground point cloud data and background point cloud data by partitioning the initial point cloud data, wherein the foreground point cloud data at least includes the target object. Next, the foreground initial scale of the foreground point cloud data is obtained, the foreground scale reference value is obtained based on the foreground initial scale, and the foreground point cloud data is compressed using the foreground encoder to obtain foreground point cloud compressed data. And the background initial scale of the background point cloud data is obtained, the background scale reference value is obtained based on the background initial scale, and the background point cloud data is compressed using the background encoder to obtain background point cloud compressed data, wherein the background scale reference value is less than the foreground scale reference value. Finally, the foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain fused point cloud data, and the fused point cloud data is input into the point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In the embodiment of the present application, the initial point cloud data is first partitioned to obtain foreground point cloud data containing the target object and background point cloud data not containing the target object. Next, the background scale reference value is set smaller than the foreground scale reference value to compress the background point cloud data with a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the amount of data in the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result.

[0146] The present application also provides a point cloud data processing device, which can implement the above point cloud data processing method. Fig.14 , the device comprises:

[0147] The point cloud segmentation module 1410 is used to partition the initial point cloud data to obtain foreground point cloud data and background point cloud data, where the foreground point cloud data at least includes the target object.

[0148] The foreground compression module 1420 is used to obtain the foreground initial scale of the foreground point cloud data, obtain the foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data.

[0149] Background compression module 1430: used to obtain the background initial scale of the background point cloud data, obtain the background scale reference value based on the background initial scale, compress the background point cloud data using the background encoder to obtain background point cloud compressed data, and the background scale reference value is smaller than the foreground scale reference value.

[0150] Point cloud reconstruction module 1440: used to decode the foreground point cloud compressed data and the background point cloud compressed data respectively to obtain the foreground point cloud decoded data and the background point cloud decoded data, splice the foreground point cloud decoded data and the background point cloud decoded data to obtain fused point cloud data, input the fused point cloud data into the point cloud reconstruction model for data reconstruction, and obtain reconstructed point cloud data.

[0151] The specific implementation of the point cloud data processing device of this embodiment is basically consistent with the specific implementation of the above-mentioned point cloud data processing method, and will not be repeated here.

[0152] The present application also provides an electronic device, including:

[0153] at least one memory;

[0154] at least one processor;

[0155] at least one program;

[0156] The program is stored in the memory, and the processor executes the at least one program to implement the point cloud data processing method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0157] See also Fig.15 , Fig.15 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0158] The processor 1501 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0159] The memory 1502 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1502, and the processor 1501 calls and executes the point cloud data processing method of the embodiment of this application;

[0160] Input / output interface 1503, used to implement information input and output;

[0161] Communication interface 1504, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0162] A bus 1505 that transmits information between the various components of the device (e.g., the processor 1501, the memory 1502, the input / output interface 1503, and the communication interface 1504);

[0163] The processor 1501 , the memory 1502 , the input / output interface 1503 and the communication interface 1504 are connected to each other in communication within the device via the bus 1505 .

[0164] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program, and the computer program implements the above-mentioned point cloud data processing method when executed by a processor.

[0165] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The point cloud data processing method, device, equipment and storage medium proposed in the embodiment of the present application obtain foreground point cloud data and background point cloud data by partitioning the initial point cloud data, wherein the foreground point cloud data at least includes the target object. Next, the foreground initial scale of the foreground point cloud data is obtained, the foreground scale reference value is obtained based on the foreground initial scale, and the foreground point cloud data is compressed using the foreground encoder to obtain foreground point cloud compressed data. And the background initial scale of the background point cloud data is obtained, the background scale reference value is obtained based on the background initial scale, and the background point cloud data is compressed using the background encoder to obtain background point cloud compressed data, wherein the background scale reference value is less than the foreground scale reference value. Finally, the foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain fused point cloud data, and the fused point cloud data is input into the point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In the embodiment of the present application, the initial point cloud data is first partitioned to obtain foreground point cloud data containing the target object and background point cloud data not containing the target object. Next, the background scale reference value is set smaller than the foreground scale reference value to compress the background point cloud data with a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the amount of data in the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result.

[0167] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0168] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0169] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0171] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0172] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0177] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A point cloud data processing method, characterized in that: include: Partitioning the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data at least includes a target object; Acquire a foreground initial scale of the foreground point cloud data, acquire a foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data; Acquire a background initial scale of the background point cloud data, acquire a background scale reference value based on the background initial scale, compress the background point cloud data using a background encoder to obtain background point cloud compressed data, wherein the background scale reference value is smaller than the foreground scale reference value; The foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced ​​to obtain fused point cloud data, and the fused point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

2. The point cloud data processing method according to claim 1, characterized in that: The foreground encoder includes at least one cascaded foreground encoding unit, and the foreground encoder is used to compress the foreground point cloud data to obtain foreground point cloud compressed data with a scale equal to the foreground scale reference value, including: Using the foreground encoding unit to encode the foreground input data to obtain a foreground encoding result, and obtaining a foreground intermediate scale parameter of the foreground encoding result, wherein the foreground input data of the first foreground encoding unit is the foreground point cloud data; If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is used as the foreground input data of the next foreground encoding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground encoding result corresponding to the foreground intermediate scale parameter is used as the foreground point cloud compression data.

3. The point cloud data processing method according to claim 2, characterized in that: The foreground encoding unit is used to encode the foreground input data to obtain a foreground encoding result, including: Downsampling the foreground input data to obtain first downsampled data; Inputting the first down-sampled data into a point cloud data processing model for data processing to obtain first point cloud compressed data; Downsampling the first point cloud compressed data to obtain second downsampled data; Channel feature processing is performed on the second down-sampled data to obtain the foreground encoding result.

4. The point cloud data processing method according to claim 3, characterized in that: The point cloud data processing model includes a downsampling extraction submodel and an upsampling extraction submodel, the downsampling extraction submodel includes a first number of cascaded downsampling extraction units, the upsampling extraction submodel includes the first number of cascaded upsampling extraction units, the downsampling extraction units and the upsampling extraction units at symmetrical positions constitute a sampling group, and the first downsampling data is input into the point cloud data processing model for data processing to obtain first point cloud compressed data, including: Inputting the first down-sampling data into the down-sampling extraction sub-model for feature extraction, obtaining first output data of each down-sampling extraction unit, wherein the first input data of a subsequent down-sampling extraction unit is the first output data of a previous down-sampling extraction unit; Input the first output data into the upsampling extraction sub-model for feature extraction, and obtain the second output data of each upsampling extraction unit, wherein the second input data of the upsampling extraction unit is the second output data of the previous upsampling extraction unit and the first output data of the downsampling extraction unit in the sampling group; A convolution operation is performed on the second output data of the last upsampling extraction unit to obtain third output data, and points with null coordinate values ​​in the third output data are deleted to obtain the first point cloud compressed data.

5. The point cloud data processing method according to claim 4, characterized in that: The downsampling extraction unit includes a sparse convolution layer and a feature extraction unit, and the feature extraction unit includes a cascaded first convolution processing submodule, a local receptive field module, and a second convolution processing submodule; the step of obtaining the first output data according to the first input data by the downsampling extraction unit includes: Using the sparse convolution layer to perform a convolution operation on the first input data to obtain sparse convolution processed data; Using the first convolution processing submodule to perform a convolution operation on the sparse convolution processing data to obtain first intermediate processing data; Using the local receptive field module to perform attention enhancement on the first intermediate processed data to obtain second intermediate processed data; The second convolution processing submodule is used to perform a convolution operation on the second intermediate processed data to obtain the first output data.

6. The point cloud data processing method according to claim 5, characterized in that: The first convolution processing submodule includes a first convolution group and a second convolution group, the first convolution group includes a plurality of first convolution layers connected in cascade, and the second convolution group includes a plurality of second convolution layers connected in cascade; the using the first convolution processing submodule to perform a convolution operation on the sparse convolution processing data to obtain first intermediate processing data includes: Inputting the sparse convolution processing data into the first convolution group to obtain the first intermediate convolution data of the last first convolution layer; Inputting the sparse convolution processed data into the second convolution group to obtain the second intermediate convolution data of the last second convolution layer; The first intermediate convolution data and the second intermediate convolution data are concatenated to obtain the first intermediate processed data.

7. The point cloud data processing method according to claim 5, characterized in that: The local receptive field module includes a second number of attention units, a local receptive field convolution layer, and a local receptive field output layer; and using the local receptive field module to perform attention enhancement on the first intermediate processed data to obtain second intermediate processed data includes: Inputting the first intermediate processed data into the attention unit for attention enhancement to obtain first local data; Accumulating the first local data to obtain second local data, and inputting the second local data into the local receptive field convolution layer for convolution operation to obtain third local data; The third local data is input into the local receptive field output layer for feature extraction to obtain the second intermediate processed data.

8. The point cloud data processing method according to claim 7, characterized in that: The attention unit includes a first attention convolution layer, a channel attention module and a second attention convolution layer; The step of inputting the first intermediate processed data into the attention unit for attention enhancement to obtain first local data includes: Inputting the first intermediate processed data into the first attention convolution layer for convolution operation to obtain first attention intermediate data; Inputting the first attention intermediate data into the channel attention module for attention calculation to obtain second attention intermediate data; The second attention intermediate data is input into the second attention convolution layer for convolution operation to obtain the first local data.

9. The point cloud data processing method according to claim 8, characterized in that: The step of inputting the third local data into the local receptive field output layer for feature extraction to obtain the second intermediate processed data comprises: Performing a softmax calculation on the third local data to obtain an output weight corresponding to the second quantity; The output weight is used to perform weighted calculation on the second attention intermediate data to obtain the second intermediate processed data.

10. The point cloud data processing method according to claim 3, characterized in that: The performing channel feature processing on the second down-sampled data to obtain the foreground encoding result includes: Performing feature fusion and then upsampling on the second downsampled data to obtain channel feature data, wherein the channel feature data includes a preset number of extended voxels corresponding to each basic voxel in the second downsampled data; The extended voxels of the same basic voxel in the channel feature data are obtained to form a voxel group, and probability estimation is performed on the extended voxels in the voxel group to obtain the foreground encoding result.

11. A point cloud data processing device, characterized in that: include: Point cloud segmentation module: used to partition the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data at least includes the target object; A foreground compression module is used to obtain a foreground initial scale of the foreground point cloud data, obtain a foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data; Background compression module: used to obtain the background initial scale of the background point cloud data, obtain the background scale reference value based on the background initial scale, compress the background point cloud data using a background encoder to obtain background point cloud compressed data, and the background scale reference value is smaller than the foreground scale reference value; Point cloud reconstruction module: used to decode the foreground point cloud compressed data and the background point cloud compressed data respectively to obtain foreground point cloud decoded data and background point cloud decoded data, splice the foreground point cloud decoded data and the background point cloud decoded data to obtain fused point cloud data, input the fused point cloud data into the point cloud reconstruction model for data reconstruction, and obtain reconstructed point cloud data.

12. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the point cloud data processing method according to any one of claims 1 to 10 when executing the computer program.

13. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the point cloud data processing method according to any one of claims 1 to 10 is implemented.

Images