Point cloud data processing method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide a point cloud data processing method, device and equipment, and a storage medium, and relate to the technical field of point cloud data processing. The method comprises the following steps: partitioning initial point cloud data to obtain foreground point cloud data and background point cloud data; setting a background scale reference value to be smaller than a foreground scale reference value; compressing the foreground point cloud data by using a foreground encoder to obtain foreground point cloud compression data; compressing the background point cloud data by using a background encoder to obtain background point cloud compression data; decoding the foreground point cloud compression data and the background point cloud compression data respectively; splicing to obtain fused point cloud data; and reconstructing the fused point cloud data to obtain reconstructed point cloud data. The background point cloud data is compressed by using a higher compression rate. The foreground point cloud data after compression has more bit streams than the background point cloud data. The key information of the target object in the foreground part is retained, the data amount of the background part is reduced, and the data accuracy and processing efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud data processing, and particularly relates to a point cloud data processing method and device, equipment and a storage medium. BACKGROUND

[0002] Point cloud compression is widely used in the field of three-dimensional data processing and transmission. Through the compression and coding process, the redundancy of point cloud data is reduced, the demand for storage space and transmission bandwidth is significantly reduced, and the efficiency of point cloud data processing is improved. Although the point cloud compression technology improves the efficiency of point cloud data processing, the point cloud compression process in the related art is difficult to simultaneously ensure the accuracy and efficiency of detection results. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a point cloud data processing method, device, equipment and storage medium to improve the data accuracy and data processing efficiency of point cloud compression results.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application provides a point cloud data processing method, comprising:

[0005] The initial point cloud data is partitioned to obtain foreground point cloud data and background point cloud data, and the foreground point cloud data at least includes a target object;

[0006] A foreground initial scale of the foreground point cloud data is obtained, a foreground scale reference value is obtained based on the foreground initial scale, a foreground encoder is used to compress the foreground point cloud data to obtain foreground point cloud compression data;

[0007] A background initial scale of the background point cloud data is obtained, a background scale reference value is obtained based on the background initial scale, a background encoder is used to compress the background point cloud data to obtain background point cloud compression data, and the background scale reference value is smaller than the foreground scale reference value;

[0008] The foreground point cloud compression data and the background point cloud compression data are respectively decoded to obtain foreground point cloud decoding data and background point cloud decoding data, the foreground point cloud decoding data and the background point cloud decoding data are spliced to obtain fused point cloud data, and the fused point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

[0009] In some embodiments, the foreground encoder includes at least one cascaded foreground encoding unit, and the use of the foreground encoder to compress the foreground point cloud data to obtain foreground point cloud compression data with a scale of the foreground scale reference value comprises:

[0010] The foreground input data is encoded by using the foreground coding unit to obtain a foreground coding result, and a foreground intermediate scale parameter of the foreground coding result is obtained, wherein the foreground input data of the first foreground coding unit is the foreground point cloud data;

[0011] If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is taken as the foreground input data of the next foreground coding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground coding result corresponding to the foreground intermediate scale parameter is taken as the foreground point cloud compression data.

[0012] In some embodiments, the foreground input data is encoded by using the foreground coding unit to obtain a foreground coding result, including:

[0013] The foreground input data is down-sampled to obtain first down-sampled data;

[0014] The first down-sampled data is input into a point cloud data processing model for data processing to obtain first point cloud compression data;

[0015] The first point cloud compression data is down-sampled to obtain second down-sampled data;

[0016] The second down-sampled data is processed by using a channel feature to obtain the foreground coding result.

[0017] In some embodiments, the point cloud data processing model includes a down-sampling extraction sub-model and an up-sampling extraction sub-model, the down-sampling extraction sub-model includes a first number of down-sampling extraction units connected in cascade, the up-sampling extraction sub-model includes the first number of up-sampling extraction units connected in cascade, the down-sampling extraction unit and the up-sampling extraction unit at a symmetrical position form a sampling group, and the first down-sampled data is input into the point cloud data processing model for data processing to obtain first point cloud compression data, including:

[0018] The first down-sampled data is input into the down-sampling extraction sub-model for feature extraction to obtain first output data of each down-sampling extraction unit, and first input data of a next down-sampling extraction unit is the first output data of a previous down-sampling extraction unit;

[0019] The first output data is input into the up-sampling extraction sub-model for feature extraction to obtain second output data of each up-sampling extraction unit, and second input data of the up-sampling extraction unit is the second output data of a previous up-sampling extraction unit and the first output data of the down-sampling extraction unit in the sampling group;

[0020] The second output data of the last upsampling extraction unit is subjected to a convolution operation to obtain third output data, and the points with coordinate null values in the third output data are deleted to obtain the first point cloud compression data.

[0021] In some embodiments, the downsampling extraction unit comprises a sparse convolution layer and a feature extraction unit, the feature extraction unit comprising a first convolution processing sub-module, a local receptive field module and a second convolution processing sub-module in cascade; the step of obtaining the first output data from the first input data by the downsampling extraction unit comprises:

[0022] The first input data is subjected to a convolution operation by the sparse convolution layer to obtain sparse convolution processing data;

[0023] The sparse convolution processing data is subjected to a convolution operation by the first convolution processing sub-module to obtain first intermediate processing data;

[0024] The first intermediate processing data is subjected to attention enhancement by the local receptive field module to obtain second intermediate processing data;

[0025] The second intermediate processing data is subjected to a convolution operation by the second convolution processing sub-module to obtain the first output data.

[0026] In some embodiments, the first convolution processing sub-module comprises a first convolution group and a second convolution group, the first convolution group comprising a plurality of first convolution layers in cascade, and the second convolution group comprising a plurality of second convolution layers in cascade; the step of subjecting the sparse convolution processing data to a convolution operation by the first convolution processing sub-module to obtain first intermediate processing data comprises:

[0027] The sparse convolution processing data is input into the first convolution group to obtain first intermediate convolution data of the last first convolution layer;

[0028] The sparse convolution processing data is input into the second convolution group to obtain second intermediate convolution data of the last second convolution layer;

[0029] The first intermediate convolution data and the second intermediate convolution data are spliced to obtain the first intermediate processing data.

[0030] In some embodiments, the local receptive field module comprises a second number of attention units, a local receptive field convolution layer and a local receptive field output layer; the step of subjecting the first intermediate processing data to attention enhancement by the local receptive field module to obtain second intermediate processing data comprises:

[0031] inputting the first intermediate processing data into the attention units respectively for attention enhancement to obtain first local data;

[0032] accumulating the first local data to obtain second local data, and inputting the second local data into the local receptive field convolution layer for convolution operation to obtain third local data;

[0033] inputting the third local data into the local receptive field output layer for feature extraction to obtain the second intermediate processing data.

[0034] In some embodiments, the attention unit includes a first attention convolution layer, a channel attention module, and a second attention convolution layer; and the inputting the first intermediate processing data into the attention units respectively for attention enhancement to obtain first local data includes:

[0035] inputting the first intermediate processing data into the first attention convolution layer for convolution operation to obtain first attention intermediate data;

[0036] inputting the first attention intermediate data into the channel attention module for attention calculation to obtain second attention intermediate data;

[0037] inputting the second attention intermediate data into the second attention convolution layer for convolution operation to obtain the first local data.

[0038] In some embodiments, the inputting the third local data into the local receptive field output layer for feature extraction to obtain the second intermediate processing data includes:

[0039] performing softmax calculation on the third local data to obtain output weights corresponding to the second number;

[0040] performing weighted calculation on the second attention intermediate data by using the output weights to obtain the second intermediate processing data.

[0041] In some embodiments, the processing the second down-sampling data to obtain the foreground encoding result includes:

[0042] performing feature fusion on the second down-sampling data and then performing up-sampling to obtain channel feature data, wherein the channel feature data includes a preset number of extended voxels corresponding to each basic voxel in the second down-sampling data;

[0043] obtaining a voxel group composed of the extended voxels of the same basic voxel in the channel feature data, and performing probability estimation on the extended voxels in the voxel group to obtain the foreground encoding result.

[0044] To achieve the above object, a second aspect of the embodiment of the present application provides a point cloud data processing apparatus, comprising:

[0045] The point cloud segmentation module is configured to segment the initial point cloud data to obtain foreground point cloud data and background point cloud data, and the foreground point cloud data at least includes a target object;

[0046] The foreground compression module is configured to obtain a foreground initial scale of the foreground point cloud data, obtain a foreground scale reference value based on the foreground initial scale, compress the foreground point cloud data by using a foreground encoder, and obtain foreground point cloud compressed data.

[0047] The background compression module is configured to obtain a background initial scale of the background point cloud data, obtain a background scale reference value based on the background initial scale, compress the background point cloud data by using a background encoder, and obtain background point cloud compressed data, wherein the background scale reference value is smaller than the foreground scale reference value.

[0048] The point cloud reconstruction module is configured to decode the foreground point cloud compressed data and the background point cloud compressed data respectively to obtain foreground point cloud decoded data and background point cloud decoded data, splice the foreground point cloud decoded data and the background point cloud decoded data to obtain fused point cloud data, input the fused point cloud data into a point cloud reconstruction model for data reconstruction, and obtain reconstructed point cloud data.

[0049] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0050] To achieve the above object, a fourth aspect of the embodiment of the present application provides a storage medium, which is a storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0051] The point cloud data processing method, apparatus, device, and storage medium proposed in this application involve partitioning initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data includes at least the target object. Next, the initial foreground scale of the foreground point cloud data is obtained, a foreground scale reference value is obtained based on the initial foreground scale, and the foreground point cloud data is compressed using a foreground encoder to obtain compressed foreground point cloud data. Simultaneously, the initial background scale of the background point cloud data is obtained, a background scale reference value is obtained based on the initial background scale, and the background point cloud data is compressed using a background encoder to obtain compressed background point cloud data, wherein the background scale reference value is smaller than the foreground scale reference value. Finally, the compressed foreground point cloud data and the compressed background point cloud data are decoded separately to obtain decoded foreground point cloud data and decoded background point cloud data. The decoded foreground point cloud data and the decoded background point cloud data are concatenated to obtain fused point cloud data. The fused point cloud data is then input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In this application embodiment, the initial point cloud data is first partitioned to obtain foreground point cloud data containing the target object and background point cloud data not containing the target object. Next, a background scale reference value is set smaller than the foreground scale reference value to compress the background point cloud data using a higher compression ratio. This results in the compressed foreground point cloud data having a larger bitstream than the background point cloud data. In subsequent point cloud data processing, this method not only preserves key information about the target object in the foreground but also reduces the amount of data in the background, thereby improving the accuracy and processing efficiency of the point cloud compression results. Attached Figure Description

[0052] Figure 1 This is a flowchart of the point cloud data processing method provided in the embodiments of this application.

[0053] Figure 2 This is a schematic diagram illustrating the partitioning of foreground point cloud data and background point cloud data provided in the embodiments of this application.

[0054] Figure 3 This is a schematic diagram illustrating the compression of foreground point cloud data and background point cloud data provided in an embodiment of this application.

[0055] Figure 4 This is a flowchart illustrating how a foreground encoding unit encodes foreground input data to obtain a foreground encoding result, as provided in an embodiment of this application.

[0056] Figure 5 This is a schematic diagram illustrating the encoding principle of the foreground encoding unit and the Beijing encoding unit provided in the embodiments of this application.

[0057] Figure 6 This is a schematic diagram of the model structure of the point cloud data processing model provided in the embodiments of this application.

[0058] Figure 7 FIG. 1 is a flowchart of a process for inputting first down-sampling data into a point cloud data processing model for data processing to obtain first point cloud compressed data, according to an embodiment of the present application.

[0059] Figure 8 FIG. 2 is a structural schematic diagram of a down-sampling extraction unit, according to an embodiment of the present application.

[0060] Figure 9 FIG. 3 is a flowchart of obtaining first output data from first input data by the down-sampling extraction unit, according to an embodiment of the present application.

[0061] Figure 10 FIG. 4 is a structural schematic diagram of a first convolution processing sub-module, according to an embodiment of the present application.

[0062] Figure 11 FIG. 5 is a structural schematic diagram of a local receptive field module, according to an embodiment of the present application.

[0063] Figure 12 FIG. 6 is a schematic diagram of testing the influence of different scale factors on detection tasks and compression tasks in a test scene, according to an embodiment of the present application.

[0064] Figure 13 FIG. 7 is a schematic diagram of compression performance in a test scene, according to an embodiment of the present application.

[0065] Figure 14 FIG. 8 is a structural block diagram of a point cloud data processing apparatus, according to another embodiment of the present application.

[0066] Figure 15 FIG. 9 is a hardware structural schematic diagram of an electronic device, according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0068] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0070] First, the terms involved in the present application are analyzed:

[0071] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0072] Point cloud compression is widely used in the field of three-dimensional data processing and transmission. Through the compression and coding process, the redundancy of point cloud data is reduced, the demand for storage space and transmission bandwidth is significantly reduced, thereby improving the efficiency of point cloud data processing. In high-precision application scenarios such as autonomous driving and virtual reality, point cloud data is often used to construct accurate three-dimensional models to support complex visual detection tasks.

[0073] Although the point cloud compression technology improves the processing efficiency of point cloud data, if a higher compression rate is selected in the point cloud compression method in the related art, more geometric information will be lost, and vice versa, if a lower compression rate is selected, the amount of point cloud data is large, which affects the data processing efficiency. Therefore, it is difficult to simultaneously ensure the detection result accuracy and detection efficiency.

[0074] Based on this, the embodiments of the present application provide a point cloud data processing method, device, equipment and storage medium. First, the initial point cloud data is partitioned to obtain foreground point cloud data containing a target object and background point cloud data not containing the target object. Next, the background scale reference value is set to be less than the foreground scale reference value, so as to compress the background point cloud data with a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the data amount of the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result.

[0075] The embodiments of the present application provide a point cloud data processing method, device, equipment and storage medium, which are specifically described by the following embodiments. First, the point cloud data processing method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0077] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0078] The point cloud data processing method provided by the embodiments of the present application relates to the technical field of point cloud data processing. The point cloud data processing method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be a computer program running in a terminal or a server. For example, the computer program can be a native program or a software module in the operating system; it can be a native application program (APP), that is, a program that needs to be installed in the operating system to run, such as a client supporting point cloud data processing, or a small program that only needs to be downloaded into a browser environment to run; it can also be a small program that can be embedded into any APP. In short, the above computer program can be any form of application program, module or plug-in. Among them, the terminal communicates with the server through a network. The point cloud data processing method can be executed by the terminal or the server, or cooperatively executed by the terminal and the server.

[0079] In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart watch, or the like. In addition, the terminal can also be a smart vehicle-mounted device. The smart vehicle-mounted device applies the point cloud data processing method of the embodiments to provide related services and improve the driving experience. The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms; or a service node in a blockchain system, the service nodes in the blockchain system form a peer-to-peer (P2P) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP) protocol. The terminal and the server can be connected through a communication connection mode such as Bluetooth, universal serial bus (USB), or a network, and the embodiments are not limited herein.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0082] The point cloud data processing method in the embodiments of the present application is described below.

[0083] Figure 1 is an optional flowchart of the point cloud data processing method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to steps 110 to 140. It can be understood that the embodiments of the present application do not make specific limitations on the order of steps 110 to 140 in Figure 1 Steps 110 to 140 in can be adjusted in order or some steps can be reduced or added according to actual needs.

[0084] Step 110: partitioning the initial point cloud data to obtain foreground point cloud data and background point cloud data.

[0085] In an embodiment, the initial point cloud data includes a target object and related background, so the embodiments of the present application first partition the initial point cloud data to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data includes the target object, and the background point cloud data does not include the target object, so the foreground point cloud data is considered as a region of interest. For the region of interest, the corresponding geometric information should be preserved as much as possible in the point cloud compression process, so as to improve the task accuracy in visual tasks such as detection and recognition. The data information in the background point cloud data is relatively redundant, which can increase the compression rate, thereby reducing the amount of data to be processed and improving the data processing efficiency. Generally, the data amount of the foreground point cloud data is much smaller than that of the background point cloud data.

[0086] In an embodiment, refer to Figure 2 , Figure 2is a schematic diagram of partitioning foreground point cloud data and background point cloud data provided by an embodiment of the present application. First, the initial point cloud data is quantized to convert floating-point coordinates into positive integer coordinates. Then, the quantized data is input into a point cloud segmentation network for segmentation. Once the segmentation is complete, all point clouds containing target objects are classified as foreground point cloud data, while point clouds not containing target objects are classified as background point cloud data. The point cloud segmentation network here can be a spatio-temporal ConvNets.

[0087] Next, the respective encoding processes of the foreground point cloud data and the background point cloud data are described.

[0088] Step 120: Obtain a foreground initial scale of the foreground point cloud data, obtain a foreground scale reference value based on the foreground initial scale, compress the foreground point cloud data using a foreground encoder to obtain foreground point cloud compressed data.

[0089] In an embodiment, different point cloud data has different geometric accuracy, so the embodiment of the present application first obtains scale data representing geometric accuracy. For foreground point cloud data, a specific foreground initial scale can be obtained, for example, according to the density or sampling interval of the point cloud of the foreground point cloud data.

[0090] Next, a foreground scale reference value is obtained based on the foreground initial scale, wherein the foreground scale reference value is used to limit the compression rate of point cloud compression, that is, the scale of the foreground point cloud compressed data after compression is close to the foreground scale reference value. Since the embodiment of the present application needs to preserve the geometric features of the foreground part, the foreground scale reference value can be selected to be close to the foreground initial scale. For example, if the foreground initial scale is N1, N1-2 can be selected as the foreground scale reference value M1.

[0091] In an embodiment, after the foreground scale reference value is determined, the point cloud compression process is started. Referring to Figure 3 , Figure 3 is a schematic diagram of compressing foreground point cloud data and background point cloud data provided by an embodiment of the present application.

[0092] Since the foreground encoder includes at least one cascaded foreground encoding unit, the foreground point cloud data is compressed by the foreground encoder to obtain foreground point cloud compressed data. Figure 3The compression process of the middle foreground point cloud data is specifically described as follows: the foreground input data is encoded by using the foreground coding unit to obtain a foreground coding result, and a foreground intermediate scale parameter of the foreground coding result is obtained. The foreground input data of the first foreground coding unit is the foreground point cloud data. If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is taken as the foreground input data of the next foreground coding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground coding result corresponding to the foreground intermediate scale parameter is taken as the foreground point cloud compression data.

[0093] That is, the foreground encoder includes a plurality of cascaded foreground coding units. In actual application, because the foreground initial scale of the specific foreground point cloud data is determined according to actual conditions, it cannot be determined which foreground coding result output by the foreground coding unit can be used as the final foreground point cloud compression data. Therefore, the data is processed step by step, and then the foreground intermediate scale parameter of the foreground coding result of each foreground coding unit is judged. If the foreground intermediate scale parameter meets the requirement of the foreground scale reference value, it is considered that the foreground intermediate scale parameter matches the foreground scale reference value when the foreground intermediate scale parameter is slightly greater than or equal to the foreground scale reference value, and the foreground coding result corresponding to the foreground intermediate scale parameter is output as the foreground point cloud compression data.

[0094] In an embodiment, referring to Figure 4 , Figure 4 is a flowchart provided by the embodiment of the application, which utilizes the foreground coding unit to encode the foreground input data to obtain a foreground coding result, and specifically includes the following steps:

[0095] Step 410: downsampling the foreground input data to obtain first downsampled data.

[0096] In an embodiment, referring to Figure 5 , Figure 5 is an encoding principle schematic diagram of the foreground coding unit and the Beijing coding unit provided by the embodiment of the application. As can be seen, the foreground input data is sequentially subjected to the first downsampling, data processing, second downsampling and channel feature processing steps in the foreground coding unit to obtain the foreground coding result until the foreground intermediate scale parameter matches the foreground scale reference value. First, the foreground input data is downsampled to obtain the first downsampled data.

[0097] Step 420: inputting the first downsampled data into the point cloud data processing model for data processing to obtain first point cloud compression data.

[0098] In an embodiment, combined withFigure 5 Next, the first down-sampling data is input into the point cloud data processing model for data processing to obtain first point cloud compressed data. This process is a lossy compression process.

[0099] With reference to Figure 6 , Figure 6 is a model structure diagram of the point cloud data processing model provided in the embodiments of the present application. Figure 6 The point cloud data processing model comprises a down-sampling extraction sub-model and an up-sampling extraction sub-model. The down-sampling extraction sub-model comprises a first number of down-sampling extraction units connected in cascade. In the diagram, the first number is 5, which does not mean that the first number can only be 5. Similarly, the up-sampling extraction sub-model comprises a first number of up-sampling extraction units connected in cascade. The down-sampling extraction unit and the up-sampling extraction unit at a symmetric position form a sampling group. Here, the symmetric position refers to axial symmetry. For example, the order of the down-sampling extraction units is 1, 2, 3, 4, and 5, and the order of the up-sampling extraction units is 1, 2, 3, 4, and 5. In this case, the first down-sampling extraction unit and the fifth up-sampling extraction unit are at a symmetric position and form a sampling group, the second down-sampling extraction unit and the fourth up-sampling extraction unit are at a symmetric position and form a sampling group, and so on.

[0100] In an embodiment, based on the structure of the point cloud data processing model, with reference to Figure 6 , Figure 7 , Figure 7 is a flowchart of inputting the first down-sampling data into the point cloud data processing model for data processing to obtain first point cloud compressed data, and comprises the following steps:

[0101] Step 710: input the first down-sampling data into the down-sampling extraction sub-model for feature extraction to obtain first output data of each down-sampling extraction unit.

[0102] In an embodiment, the first down-sampling data is input into the down-sampling extraction sub-model for feature extraction, with reference to Figure 6 , and sequentially passes through five down-sampling extraction units for data processing. Assume that the input data of each down-sampling extraction unit is referred to as first input data, and the output data is referred to as first output data. In this case, the first input data of the first down-sampling extraction unit is the first down-sampling data, and the first input data of the subsequent down-sampling extraction unit is the first output data of the previous down-sampling extraction unit. At the same time, the first output data of each down-sampling extraction unit needs to be obtained.

[0103] In an embodiment, with reference to Figure 8 , Figure 8is a structural schematic diagram of a down-sampling extraction unit provided by the embodiment. Figure 8 The down-sampling extraction unit comprises a sparse convolution layer (S shown in the figure) and a feature extraction unit (FE shown in the figure), wherein the feature extraction unit comprises a first convolution processing submodule, a local receptive field module and a second convolution processing submodule in cascade.

[0104] In an embodiment, the down-sampling extraction unit comprises a sparse convolution layer (S shown in the figure) and a feature extraction unit (FE shown in the figure), wherein the feature extraction unit comprises a first convolution processing submodule, a local receptive field module and a second convolution processing submodule in cascade. Figure 8 Figure 9 , Figure 9 is a flowchart of the down-sampling extraction unit provided by the embodiment of the present application for obtaining first output data according to first input data, comprising the following steps:

[0105] Step 910: performing convolution operation on the first input data by using the sparse convolution layer to obtain sparse convolution processing data.

[0106] In an embodiment, the sparse convolution layer is specially used for processing sparse data, and the sparse data refers to data in which most elements are zero or close to zero. Unlike the convolution layer which calculates all elements of the input, the sparse convolution layer only needs to calculate the non-zero elements, which can improve the calculation efficiency. The specific process is as follows: the first input data is converted into a sparse representation form, only the non-zero elements and their position information are retained. Next, according to the sparse representation form, only the non-zero elements are subjected to convolution operation, and finally the convolution result is recombined into complete output data to obtain the sparse convolution processing data.

[0107] Step 920: performing convolution operation on the sparse convolution processing data by using the first convolution processing submodule to obtain first intermediate processing data.

[0108] In an embodiment, the down-sampling extraction unit comprises a sparse convolution layer (S shown in the figure) and a feature extraction unit (FE shown in the figure), wherein the feature extraction unit comprises a first convolution processing submodule, a local receptive field module and a second convolution processing submodule in cascade. Figure 10 , Figure 10 is a structural schematic diagram of the first convolution processing submodule provided by the embodiment of the present application. Figure 10 The first convolution processing submodule comprises a first convolution group and a second convolution group, wherein the first convolution group comprises a plurality of first convolution layers in cascade, and three first convolution layers are taken as an example in the figure, and the convolution kernel sizes of the illustrated first convolution layers are 1x1x1, 3x3x3 and 1x1x1 respectively. The second convolution group comprises a plurality of second convolution layers in cascade, and two second convolution layers are taken as an example in the figure, and the convolution kernel sizes of the illustrated second convolution layers are 3x3x3 and 3x3x3 respectively. It can be understood that the number and the convolution kernel size of the first convolution layers and the second convolution layers respectively can be set according to actual requirements, and the embodiment only illustrates this.

[0109] In an embodiment, the down-sampling extraction unit comprises a sparse convolution layer (S shown in the figure) and a feature extraction unit (FE shown in the figure), wherein the feature extraction unit comprises a first convolution processing submodule, a local receptive field module and a second convolution processing submodule in cascade. Figure 10 ​The process of obtaining the first intermediate processing data by using the first convolution processing submodule to perform convolution operation on the sparse convolution processing data based on the structural basis is specifically described as follows: the sparse convolution processing data is input into the first convolution group to obtain the first intermediate convolution data of the last first convolution layer, and at the same time, the sparse convolution processing data is input into the second convolution group to obtain the second intermediate convolution data of the last second convolution layer, and then the first intermediate convolution data and the second intermediate convolution data are spliced to obtain the first intermediate processing data.

[0110] Step 930: performing attention enhancement on the first intermediate processing data by using the local receptive field module to obtain second intermediate processing data.

[0111] In an embodiment, referring to Figure 11 , Figure 11 is a structural schematic diagram of the local receptive field module provided in the embodiment of the application. Figure 11 The local receptive field module includes a second number of attention units, a local receptive field convolution layer and a local receptive field output layer, wherein the second number is taken as an example of 4, which does not represent a limitation. Figure 11 The attention unit includes a first attention convolution layer, a channel attention module and a second attention convolution layer.

[0112] Therefore, in combination with the structure of the local receptive field module, Figure 11 the process of performing attention enhancement on the first intermediate processing data by using the local receptive field module to obtain second intermediate processing data is described as follows:

[0113] First, the first intermediate processing data is input into the attention unit for attention enhancement to obtain first local data, that is, the same first intermediate processing data is input into the attention unit for parallel calculation. At this time, in combination with Figure 11 in the processing process of the attention unit, the first intermediate processing data is input into the first attention convolution layer for convolution operation to obtain first attention intermediate data, the first attention intermediate data is input into the channel attention module for attention calculation to obtain second attention intermediate data, and finally the second attention intermediate data is input into the second attention convolution layer for convolution operation to obtain the first local data.

[0114] In an embodiment, the convolution kernel sizes of the first attention convolution layer are 3x3x3, 5x5x5, 7x7x7, 9x9x9 in sequence of the attention units. In addition, the channel attention module is an efficient channel attention (ECA) which generates channel attention through a fast one-dimensional convolution, and the convolution kernel size thereof can be adaptively determined by a nonlinear mapping of the channel dimension. Finally, the convolution kernel sizes of the second attention convolution layer of different attention units are all 1x1x1. It can be seen that different convolution kernel sizes are used in the attention units to improve the feature extraction performance.

[0115] Next, after each attention unit obtains the corresponding first local data, the first local data is accumulated to obtain second local data. After obtaining the second local data, it is input into the local receptive field convolution layer for convolution operation to obtain third local data, wherein the local receptive field convolution layer is a convolution layer with a convolution kernel size of 1x1x1.

[0116] In an embodiment, the third local data is input into the local receptive field output layer for feature extraction to obtain second intermediate processing data. In the local receptive field output layer, softmax calculation is performed on the third local data to obtain output weights corresponding to the second quantity, wherein the output weights and the second attention intermediate data have a corresponding relationship, and finally the output weights are used to perform weighted calculation on the second attention intermediate data to obtain the second intermediate processing data.

[0117] Referring to Figure 11 , assuming that the first attention intermediate data in the four attention units are represented as x1, x2, x3 and x4, the corresponding second attention intermediate data are represented as x1', x2', x3' and x4', and the first local data are represented as w1, w2, w3 and w4. After accumulation, the third local data obtained by inputting into the local receptive field convolution layer for convolution operation is represented as w. After feature extraction by the local receptive field output layer, the second intermediate processing data are represented as f1, f2, f3 and f4, wherein f1 and x1' correspond, f2 and x2' correspond, f3 and x3' correspond, and f4 and x4' correspond. After weighting, the second intermediate processing data are represented as x.

[0118] Step 940: performing convolution operation on the second intermediate processing data by using the second convolution processing sub-module to obtain first output data.

[0119] In an embodiment, the second convolution processing submodule is the same as the first convolution processing submodule in structure, and the corresponding convolution kernel size can be set according to actual requirements, or the same model structure as the first convolution processing submodule can be adopted. After obtaining the second intermediate processing data in the embodiment of the application, the second convolution processing submodule is used to perform convolution operation on the second intermediate processing data, and the first output data can be obtained.

[0120] Step 720: inputting the first output data into the up-sampling extraction sub-model to perform feature extraction, and obtaining the second output data of each up-sampling extraction unit.

[0121] In an embodiment, referring to Figure 6 , the up-sampling extraction sub-model includes five up-sampling extraction units that form a sampling group with the down-sampling extraction units. Since the up-sampling extraction units are in a cascaded relationship, it is assumed that the input data of each up-sampling extraction unit is referred to as second input data, and the output data is referred to as second output data. At this time, the second input data of the subsequent up-sampling extraction unit is the second output data of the previous up-sampling extraction unit and the first output data of the down-sampling extraction unit in the sampling group.

[0122] For example, for the first up-sampling extraction unit, it forms a sampling group with the fifth down-sampling extraction unit. At this time, theoretically, the second input data of the first up-sampling extraction unit includes the first output data of the fifth down-sampling extraction unit and the second output data of the previous up-sampling extraction unit. Since the first up-sampling extraction unit is the first module of the down-sampling extraction sub-model, its second input data only includes the first output data of the fifth down-sampling extraction unit. For the fifth up-sampling extraction unit, it forms a sampling group with the first down-sampling extraction unit. At this time, the second input data of the fifth up-sampling extraction unit includes the first output data of the first down-sampling extraction unit and the second output data of the fourth up-sampling extraction unit. In this way, the second input data of each down-sampling extraction unit can be determined, and the corresponding second output data can be calculated.

[0123] In an embodiment, referring to Figure 6 , the down-sampling extraction unit is similar to the up-sampling extraction unit and includes a deconvolution layer and a feature extraction unit (FE shown in the figure). The deconvolution layer includes two different types of deconvolution layers, for example, the first type is a MinkowskiConvolutionTranspose deconvolution layer, that is, TS shown in Figure 6 , and the second type is a MinkowskiGenerativeConvolutionTranspose deconvolution layer, that is, Figure 6The two deconvolution layers are modules in the MinkowskiEngine library and are used to perform different types of deconvolution operations, as shown in FIG. 6. Refer to Figure 6 In order, the combination types of the upsampling extraction units are: TS+FE, TS+FE, TS+FE, GTS+FE, TS+FE.

[0124] Step 730: Perform a convolution operation on the second output data of the last upsampling extraction unit to obtain third output data, and delete the point cloud with a coordinate null value in the third output data to obtain first point cloud compression data.

[0125] In an embodiment, referring to Figure 6 Perform a convolution operation on the second output data of the last upsampling extraction unit to obtain third output data, and the convolution operation can be a sparse convolution operation. Next, delete the point with a coordinate null value in the third output data to obtain the first point cloud compression data.

[0126] Step 430: Perform downsampling on the first point cloud compression data to obtain second down-sampling data.

[0127] In an embodiment, in combination Figure 5 After obtaining the first point cloud compression data, perform a second downsampling operation thereon to obtain corresponding second down-sampling data, for further compression.

[0128] Step 440: Perform channel feature processing on the second down-sampling data to obtain foreground encoding results.

[0129] In an embodiment, perform channel feature processing on the second down-sampling data to learn the spatial correlation of the point cloud and perform voxel probability estimation. The process of performing channel feature processing on the second down-sampling data to obtain the foreground encoding results is described as follows:

[0130] First, the second down-sampling data is fused and then up-sampled to obtain channel feature data. Among them, the input second down-sampling data is processed and fused by multiple layers through deep feature aggregation. First, initial features are extracted from the second down-sampling data, which may include basic information such as spatial position and local geometric shape. The extracted initial features are preliminarily fused through connection operation or addition operation. Next, the preliminarily fused features are processed by multiple layers, and in each layer of processing, the features are further extracted through convolution operation and nonlinear activation function. After multiple layers of processing, the features at different levels are integrated together to form the final channel fusion data. Next, the channel fusion data is up-sampled, and the channel fusion data is expanded to a preset number of smaller voxels, such as 8, through a convolution operation with a convolution kernel size of 2x2x2, thereby increasing the size and resolution of the feature map to obtain channel feature data. At this time, assuming that the voxels in the channel feature data are called expanded voxels, and the voxels in the second down-sampling data are called basic voxels, after the above process, each basic voxel becomes 8 smaller expanded voxels, and these expanded voxels observe the basic voxel from different perspectives, each perspective corresponds to a specific feature representation, and can represent the multi-dimensional information of the point cloud data.

[0131] Next, the expanded voxels of the same basic voxel in the channel feature data form a voxel group, assuming that the number of basic voxels is n, at this time there are n voxel groups, and each voxel group contains 8 expanded voxels. The expanded voxels in the voxel group are estimated to obtain the foreground encoding result. Among them, different voxel groups can be parallelly estimated. Specifically, for each voxel group, the expanded voxels are sequentially subjected to multiple stage probability estimation, and the occupancy probability of the expanded voxels is calculated, for example, the number of stages is 8. In each stage, the probability of each expanded voxel belonging to a specific class is calculated according to the channel feature data, and the specific class is set according to the actual situation. In each stage, the occupancy probability of the expanded voxels is updated based on the result calculated in the previous stage, and then the channel feature data is updated based on the occupancy probability. After multiple iterations, the final channel feature data is obtained as the foreground encoding result.

[0132] Step 130: obtaining a background initial scale of the background point cloud data, obtaining a background scale reference value based on the background initial scale, compressing the background point cloud data by using a background encoder, and obtaining background point cloud compression data, wherein the background scale reference value is smaller than the foreground scale reference value.

[0133] In an embodiment, the background initial scale of the background point cloud data is obtained in the same way as the foreground initial scale, and then a background scale reference value is obtained based on the background initial scale, wherein the background scale reference value is used to limit the compression rate of the point cloud compression, that is, the scale of the background point cloud compressed data after compression is close to the background scale reference value. Since the embodiment of the present application needs to reduce the data amount of the background part, the foreground scale reference value can be much smaller than the value of the background initial scale. For example, the background initial scale is N2, and N2-9 can be selected as the background scale reference value M2.

[0134] Next, referring to Figure 3 , the background encoder includes at least one cascaded background encoding unit. The compression process of the background point cloud data is specifically described as follows: the background input data is encoded by using the background encoding unit to obtain a background encoding result, and a background intermediate scale parameter of the background encoding result is obtained. The background input data of the first background encoding unit is the background point cloud data. If the background intermediate scale parameter does not match the background scale reference value, the background intermediate scale parameter is taken as the background input data of the next background encoding unit until the background intermediate scale parameter matches the background scale reference value, and the background encoding result corresponding to the background intermediate scale parameter is taken as the background point cloud compressed data.

[0135] In an embodiment, the background encoding unit and the foreground encoding unit are similar in structure, and the difference between them is mainly that the background scale reference value is different from the foreground scale reference value. The processing procedure is consistent, and the details are described in Figure 5 In the background encoding unit, the background input data is also sequentially subjected to the first downsampling, data processing, second downsampling, and channel feature processing to obtain the background encoding result until the background intermediate scale parameter matches the background scale reference value.

[0136] As can be seen from the above, the processing procedure of the foreground point cloud compressed data can be referred to to obtain the background point cloud compressed data.

[0137] In an embodiment, the bit numbers allocated in the foreground point cloud data compression process and the background point cloud data compression process are different. Assuming that the total bit number is R all , the bit stream of the foreground part is R foreground , and the bit stream of the background part is R background , wherein R all =R foreground +R background . In the entire compression process, more bits are allocated to the foreground part, and fewer bits are allocated to the background part.

[0138] Step 140: decode the foreground point cloud compression data and the background point cloud compression data respectively to obtain foreground point cloud decoding data and background point cloud decoding data, splice the foreground point cloud decoding data and the background point cloud decoding data to obtain fused point cloud data, and input the fused point cloud data into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

[0139] In an embodiment, the point cloud data processing method is combined with Figure 3 , the decoding process in this embodiment corresponds to the encoding process. Then, the foreground point cloud decoding data and the background point cloud decoding data are spliced to obtain fused point cloud data, and at this time, the fused point cloud data contains a foreground part and a background part. Finally, the fused point cloud data is processed based on a point cloud reconstruction model to obtain final reconstructed point cloud data. In this embodiment, the point cloud reconstruction model is consistent with the model structure of the point cloud data processing model shown in Figure 6 , and the model parameters are different according to the actual training process.

[0140] In an embodiment, after obtaining the reconstructed point cloud data, a downstream visual detection task can be performed, and the detection task is set according to actual requirements.

[0141] The test process of the point cloud data processing method provided by the embodiments of the present application is described below.

[0142] First, the related models used in the point cloud data processing method are pre-trained using the KITTI dataset. The KITTI dataset is quantized to an accuracy of 2 cm (14 bits). Then, the dataset is divided into a training set and a test set. In order to evaluate the compression performance, the bits per point (bpp) is used as an indicator of the compression ratio, and the point-to-point error (D1) and the point-to-plane error (D2) are used as distortion indicators. In the detection task, mAP@0.25 and mAP@0.5 are used as evaluation criteria. In addition, in order to perform fair experimental comparison, some models in related technologies are selected for comparison, such as G-PCC TMC13-v14, AVS PCRM-v11.0, and PCGCv2, and the point cloud data processing method in the embodiments of the present application is denoted as SparsePCGC. For the downstream detection task, Group-Free is used as the detection network, the computer is configured as NVIDIA RTX A5000, and the code is developed based on the PyTorch framework.

[0143] In an embodiment, the scale factor of the foreground part is denoted as Scalef, the scale factor of the background part is denoted as Scaleb, which is used to reflect different bit stream allocation for the foreground and the background, Bppf / Bppb represents the bit ratio between the foreground and the background regions, and Bpp represents the total bit stream. Referring to Figure 12, Figure 12 This diagram illustrates the impact of different scale factors on the detection and compression tasks in a test scenario of an embodiment of this application. (Combined with...) Figure 12 The data shows that the performance of compression and detection tasks varies under different bit conditions. For example, the highlighted data at mAP@0.25 (Scale f=2) clearly demonstrates that allocating more bits to the foreground region can maintain a high level of performance in the downstream detection task, clarifying the critical importance of the foreground region in the detection task. However, when the number of bits in the foreground region is reduced (e.g., Scale f=4), the detection task performance drops significantly, with an overall value of approximately 0.05.

[0144] Next, refer to Figure 13 , Figure 13 This is a schematic diagram of compression performance in a test scenario provided in an embodiment of this application. Figure 13 The horizontal axis represents Bpp, and the vertical axis represents the values ​​of three different evaluation metrics: point-to-point error (D1), point-to-plane error (D2), and mAP@0.25. This illustrates the combination of reconstructed point clouds obtained by encoding the foreground and background separately, used to evaluate their compression performance. Specifically, SAVD-ALL represents the compression performance without separate encoding of the foreground and background data; while SAVD-SPLIT represents the performance of the method in this embodiment after separately encoding the foreground and background data. Clearly, SAVD-SPLIT outperforms SAVD-ALL in compression performance. Furthermore, the method in this embodiment outperforms AVS PCRM and PCGCv2 in compression performance, and is comparable to G-PCC and SparsePCGC. Therefore, under the same bitrate conditions, the method in this embodiment significantly improves detection performance in the high bitrate range while maintaining good detection quality.

[0145] The point cloud data processing method provided in the embodiments of the present application comprises the following steps: initial point cloud data is partitioned to obtain foreground point cloud data and background point cloud data, wherein the foreground point cloud data at least comprises a target object; a foreground initial scale of the foreground point cloud data is obtained, a foreground scale reference value is obtained based on the foreground initial scale, a foreground encoder is used to compress the foreground point cloud data to obtain foreground point cloud compressed data; a background initial scale of the background point cloud data is obtained, a background scale reference value is obtained based on the background initial scale, a background encoder is used to compress the background point cloud data to obtain background point cloud compressed data, wherein the background scale reference value is smaller than the foreground scale reference value; the foreground point cloud compressed data and the background point cloud compressed data are decoded respectively to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced to obtain fused point cloud data, and the fused point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In the embodiments of the present application, the initial point cloud data is first partitioned to obtain the foreground point cloud data comprising the target object and the background point cloud data not comprising the target object. Next, the background scale reference value is set to be smaller than the foreground scale reference value, so that the background point cloud data is compressed by using a higher compression rate. In this way, the foreground point cloud data after compression has more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the data amount of the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result.

[0146] The embodiments of the present application also provide a point cloud data processing device, which can implement the point cloud data processing method. Figure 14 The device comprises:

[0147] The point cloud segmentation module 1410 is configured to partition the initial point cloud data to obtain the foreground point cloud data and the background point cloud data, wherein the foreground point cloud data at least comprises the target object.

[0148] The foreground compression module 1420 is configured to obtain the foreground initial scale of the foreground point cloud data, obtain the foreground scale reference value based on the foreground initial scale, and compress the foreground point cloud data by using the foreground encoder to obtain the foreground point cloud compressed data.

[0149] The background compression module 1430 is configured to obtain the background initial scale of the background point cloud data, obtain the background scale reference value based on the background initial scale, and compress the background point cloud data by using the background encoder to obtain the background point cloud compressed data, wherein the background scale reference value is smaller than the foreground scale reference value.

[0150] The point cloud reconstruction module 1440 is configured to decode the foreground point cloud compression data and the background point cloud compression data respectively to obtain foreground point cloud decoding data and background point cloud decoding data, splice the foreground point cloud decoding data and the background point cloud decoding data to obtain fused point cloud data, and input the fused point cloud data into a point cloud reconstruction model to perform data reconstruction to obtain reconstructed point cloud data.

[0151] The specific implementation of the point cloud data processing apparatus in the embodiment is basically the same as the specific implementation of the point cloud data processing method, and will not be repeated here.

[0152] The embodiment of the present application further provides an electronic device, comprising:

[0153] at least one memory;

[0154] at least one processor;

[0155] at least one program;

[0156] The program is stored in the memory, and the processor executes the at least one program to implement the point cloud data processing method provided in the embodiment of the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0157] Please refer to Figure 15 , Figure 15 The hardware structure of the electronic device of another embodiment is shown, and the electronic device comprises:

[0158] The processor 1501 can be implemented in the form of a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0159] The memory 1502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 1502 and are called and executed by the processor 1501 to implement the point cloud data processing method in the embodiments of the present application.

[0160] The input / output interface 1503 is configured to realize information input and output.

[0161] The communication interface 1504 is configured to realize communication interaction between the device and other devices, and the communication can be realized through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like).

[0162] The bus 1505 is configured to transmit information between various components (for example, the processor 1501, the memory 1502, the input / output interface 1503 and the communication interface 1504) of the device.

[0163] The processor 1501, the memory 1502, the input / output interface 1503 and the communication interface 1504 are connected to each other through the bus 1505 to realize communication connection between the device.

[0164] The embodiment of the present application further provides a storage medium, the storage medium is a storage medium, and the storage medium stores a computer program. The computer program is executed by the processor to realize the point cloud data processing method.

[0165] The memory is a non-transient storage medium, and can be used to store a non-transient software program and a non-transient computer executable program. In addition, the memory can include a high-speed random access memory, and can further include a non-transient memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transient solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0166] The point cloud data processing method, device, equipment and storage medium provided in the embodiments of the present application obtain foreground point cloud data and background point cloud data by partitioning the initial point cloud data, wherein the foreground point cloud data at least includes a target object. Next, a foreground initial scale of the foreground point cloud data is obtained, a foreground scale reference value is obtained based on the foreground initial scale, the foreground point cloud data is compressed by using a foreground encoder to obtain foreground point cloud compressed data. And a background initial scale of the background point cloud data is obtained, a background scale reference value is obtained based on the background initial scale, the background point cloud data is compressed by using a background encoder to obtain background point cloud compressed data, wherein the background scale reference value is less than the foreground scale reference value. Finally, the foreground point cloud compressed data and the background point cloud compressed data are respectively decoded to obtain foreground point cloud decoded data and background point cloud decoded data, the foreground point cloud decoded data and the background point cloud decoded data are spliced to obtain fused point cloud data, and the fused point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data. In the embodiments of the present application, the initial point cloud data is first partitioned to obtain foreground point cloud data containing a target object and background point cloud data not containing the target object. Next, the background scale reference value is set to be less than the foreground scale reference value, so that the background point cloud data is compressed by using a higher compression rate. In this way, the compressed foreground point cloud data will have more bit streams than the background point cloud data. In the subsequent point cloud data processing process, this method not only retains the key information of the target object in the foreground part, but also reduces the data amount of the background part, thereby improving the data accuracy and processing efficiency of the point cloud compression result.

[0167] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0168] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0169] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0170] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the equipment can be implemented as software, firmware, hardware and their appropriate combinations.

[0171] The terms "first", "second", "third", "fourth" and the like in the description of this application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a changeable order, sequence or arrangement, if any. Further, the terms "comprising", "having", "including", and "containing" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises, has, includes or contains a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, system, product or apparatus.

[0172] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0173] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0174] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0175] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0176] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0177] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for processing point cloud data, characterized in that, The application relates to a point cloud compression method and device. The initial point cloud data is partitioned to obtain foreground point cloud data and background point cloud data, and the foreground point cloud data at least includes a target object; A foreground initial scale of the foreground point cloud data is obtained, a foreground scale reference value is obtained based on the foreground initial scale, a foreground encoder includes at least one cascaded foreground encoding unit, foreground input data is processed by the foreground encoding unit to obtain a foreground encoding result, and a foreground intermediate scale parameter of the foreground encoding result is obtained, and the foreground input data of a first foreground encoding unit is the foreground point cloud data; If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is taken as the foreground input data of a next foreground encoding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground encoding result corresponding to the foreground intermediate scale parameter is taken as foreground point cloud compression data; A background initial scale of the background point cloud data is obtained, a background scale reference value is obtained based on the background initial scale, and the background point cloud data is compressed by a background encoder to obtain background point cloud compression data, and the background scale reference value is smaller than the foreground scale reference value; The foreground point cloud compression data and the background point cloud compression data are respectively decoded to obtain foreground point cloud decoding data and background point cloud decoding data, the foreground point cloud decoding data and the background point cloud decoding data are spliced to obtain fusion point cloud data, and the fusion point cloud data is input into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

2. The point cloud data processing method of claim 1, wherein, The foreground input data is processed by the foreground encoding unit to obtain the foreground encoding result, and the method comprises the following steps: The foreground input data is down-sampled to obtain first down-sampling data; The first down-sampling data is input into a point cloud data processing model for data processing to obtain first point cloud compression data; The first point cloud compression data is down-sampled to obtain second down-sampling data; The second down-sampling data is processed by a channel feature to obtain the foreground encoding result.

3. The point cloud data processing method of claim 2, wherein, The point cloud data processing model comprises a down-sampling extraction sub-model and an up-sampling extraction sub-model, the down-sampling extraction sub-model comprises a first number of cascaded down-sampling extraction units, the up-sampling extraction sub-model comprises a first number of cascaded up-sampling extraction units, the down-sampling extraction unit and the up-sampling extraction unit at a symmetrical position form a sampling group, and the first down-sampling data is input into the point cloud data processing model for data processing to obtain the first point cloud compression data, which comprises the following steps: The first down-sampling data is input into the down-sampling extraction sub-model for feature extraction to obtain first output data of each down-sampling extraction unit, and the first input data of a next down-sampling extraction unit is the first output data of a previous down-sampling extraction unit; inputting the first output data into the up-sampling extraction sub-model for feature extraction, obtaining second output data of each up-sampling extraction unit, and the second input data of the up-sampling extraction unit being the second output data of a previous up-sampling extraction unit and the first output data of the down-sampling extraction unit in the sampling group; performing convolution operation on the second output data of the last up-sampling extraction unit to obtain third output data, and deleting points with coordinate null values in the third output data to obtain the first point cloud compression data.

4. The point cloud data processing method of claim 3, wherein, The down-sampling extraction unit comprises a sparse convolution layer and a feature extraction unit, and the feature extraction unit comprises a cascaded first convolution processing sub-module, a local receptive field module and a second convolution processing sub-module; the step of obtaining the first output data from the first input data by the down-sampling extraction unit comprises: performing convolution operation on the first input data by the sparse convolution layer to obtain sparse convolution processing data; performing convolution operation on the sparse convolution processing data by the first convolution processing sub-module to obtain first intermediate processing data; performing attention enhancement on the first intermediate processing data by the local receptive field module to obtain second intermediate processing data; performing convolution operation on the second intermediate processing data by the second convolution processing sub-module to obtain the first output data.

5. The point cloud data processing method of claim 4, wherein, The first convolution processing sub-module comprises a first convolution group and a second convolution group, the first convolution group comprises a plurality of first convolution layers in cascade, and the second convolution group comprises a plurality of second convolution layers in cascade; the step of performing convolution operation on the sparse convolution processing data by the first convolution processing sub-module to obtain first intermediate processing data comprises: inputting the sparse convolution processing data into the first convolution group to obtain first intermediate convolution data of the last first convolution layer; inputting the sparse convolution processing data into the second convolution group to obtain second intermediate convolution data of the last second convolution layer; splicing the first intermediate convolution data and the second intermediate convolution data to obtain the first intermediate processing data.

6. The point cloud data processing method of claim 4, wherein, The local receptive field module comprises a second number of attention units, a local receptive field convolution layer and a local receptive field output layer; the step of performing attention enhancement on the first intermediate processing data by the local receptive field module to obtain second intermediate processing data comprises: inputting the first intermediate processing data into the attention units respectively for attention enhancement to obtain first local data; accumulating the first local data to obtain second local data, and inputting the second local data into the local receptive field convolution layer for convolution operation to obtain third local data; inputting the third local data into the local receptive field output layer for feature extraction to obtain the second intermediate processing data.

7. The point cloud data processing method of claim 6, wherein, The attention unit comprises a first attention convolution layer, a channel attention module and a second attention convolution layer; the step of inputting the first intermediate processing data into the attention units respectively for attention enhancement to obtain first local data comprises: The first intermediate processing data is input into the first attention convolution layer for convolution operation to obtain first attention intermediate data; The first attention intermediate data is input into the channel attention module for attention calculation to obtain second attention intermediate data; The second attention intermediate data is input into the second attention convolution layer for convolution operation to obtain the first local data.

8. The point cloud data processing method of claim 7, wherein, The third local data is input into the local receptive field output layer for feature extraction to obtain the second intermediate processing data, including: The third local data is subjected to softmax calculation to obtain output weights corresponding to the second number; The second attention intermediate data is subjected to weighted calculation using the output weights to obtain the second intermediate processing data.

9. The point cloud data processing method of claim 2, wherein, The second down-sampling data is subjected to channel feature processing to obtain the foreground encoding result, including: The second down-sampling data is subjected to feature fusion and then up-sampling to obtain channel feature data, the channel feature data including a preset number of extended voxels corresponding to each basic voxel in the second down-sampling data; The extended voxels of the same basic voxel in the channel feature data form a voxel group, and the extended voxels in the voxel group are subjected to probability estimation to obtain the foreground encoding result.

10. A point cloud data processing apparatus, characterized by comprising: including: The point cloud segmentation module is configured to segment initial point cloud data to obtain foreground point cloud data and background point cloud data, and the foreground point cloud data includes at least a target object; The foreground compression module is configured to obtain a foreground initial scale of the foreground point cloud data, obtain a foreground scale reference value based on the foreground initial scale, and obtain a foreground encoding result by encoding foreground input data using at least one cascaded foreground encoding unit, and obtain a foreground intermediate scale parameter of the foreground encoding result, wherein the foreground input data of a first foreground encoding unit is the foreground point cloud data; If the foreground intermediate scale parameter does not match the foreground scale reference value, the foreground intermediate scale parameter is used as the foreground input data of a next foreground encoding unit until the foreground intermediate scale parameter matches the foreground scale reference value, and the foreground encoding result corresponding to the foreground intermediate scale parameter is used as foreground point cloud compression data; The background compression module is configured to obtain a background initial scale of the background point cloud data, obtain a background scale reference value based on the background initial scale, and obtain background point cloud compression data by compressing the background point cloud data using a background encoder, wherein the background scale reference value is smaller than the foreground scale reference value; The point cloud reconstruction module is configured to decode the foreground point cloud compression data and the background point cloud compression data to obtain foreground point cloud decoding data and background point cloud decoding data, splice the foreground point cloud decoding data and the background point cloud decoding data to obtain fused point cloud data, and input the fused point cloud data into a point cloud reconstruction model for data reconstruction to obtain reconstructed point cloud data.

11. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the point cloud data processing method in any one of claims 1 to 9 when executing the computer program.

12. A storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the point cloud data processing method in any one of claims 1 to 9.

Images