Point cloud data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a point cloud data processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining voxelized point cloud data; constructing a convolution mapping table according to the size and the moving step length of a convolution kernel in the convolutional neural network; determining a feature input matrix corresponding to the voxelized point cloud data based on a convolution mapping table; performing deformation processing on the convolution kernel based on the feature input matrix to obtain a target convolution kernel; and performing matrix multiplication calculation on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result so as to perform task processing based on the point cloud data by using the sparse convolution operation result. According to the embodiment of the invention, when the sparse convolution operation is carried out, the sparse convolution operation result can be obtained through one-time matrix multiplication calculation, an intermediate result of multiplication accumulation does not need to be reserved in a memory, and the memory space occupied in the convolution operation process is greatly reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus, device and storage medium for processing point cloud data. Background Art

[0002] Three-dimensional information is the most intuitive information in the real physical environment. The acquisition, perception, analysis and understanding of three-dimensional information is one of the key technologies in many application fields.

[0003] At present, deep neural networks are often used to process and analyze point cloud data. One of the most commonly used operators in deep neural networks is convolution. When performing convolution operations on voxelized point clouds, the corresponding convolution kernel is also three-dimensional. Each element on the convolution kernel is iteratively multiplied and accumulated with the input point cloud features as the convolution kernel steps. However, since valid point cloud grids are very rare in voxelized point clouds, most of the grids in the space are filled with zero values. Therefore, the elements of the convolution kernel are multiplied with a large number of zero values. These invalid outputs occupy most of the space and computing power of the point cloud convolution.

[0004] In the related technology, when performing convolution operations on point cloud data on an artificial intelligence (AI) chip, each element on the convolution kernel is iteratively multiplied and accumulated with the input point cloud features as the convolution kernel steps. Therefore, the product and accumulation results need to be kept in the memory until all convolution calculations are completed, resulting in a large amount of memory space being occupied when performing convolution operations on point cloud data. Summary of the invention

[0005] The embodiments of the present disclosure provide a method and apparatus, a device and a storage medium for processing point cloud data to solve the problem of occupying a large amount of memory space when performing convolution operations on point cloud data.

[0006] According to a first aspect of an embodiment of the present disclosure, a data processing method based on point cloud data is provided, comprising:

[0007] Get voxelized point cloud data;

[0008] Based on the voxelized point cloud data, the size and moving step of the convolution kernel in the convolutional neural network, a convolution mapping table is constructed, wherein the index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, an offset position of the convolution kernel and each valid output position, and each valid input position is a position of each valid point in the voxelized point cloud data;

[0009] Based on the convolution mapping table, determining a feature input matrix corresponding to the voxelized point cloud data, wherein each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to the product of each input channel of the voxelized point cloud data and the number of elements of the convolution kernel;

[0010] Performing deformation processing on the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix;

[0011] A matrix multiplication calculation is performed on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing based on point cloud data.

[0012] In some embodiments of the present disclosure, constructing a convolution mapping table based on the voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step size includes:

[0013] Determining each of the valid input positions based on the voxelized point cloud data;

[0014] Determine the valid output positions based on the valid input positions, the size of the convolution kernel and the moving step size;

[0015] Based on the valid input positions and the valid output positions, construct a position index table, wherein the position index table includes the valid input positions and the indexes of the valid input positions, and the valid output positions and the indexes of the valid output positions;

[0016] The convolution mapping table is constructed based on the position index table and the offset position of the convolution kernel, and the index of the convolution mapping table is generated based on each valid output position.

[0017] In some embodiments of the present disclosure, constructing the convolution mapping table based on the position index table and the offset position of the convolution kernel includes:

[0018] Determine, based on the offset position of the convolution kernel, a mapping relationship between each valid output position and each valid input position;

[0019] Based on the position index table and the mapping relationship, the convolution mapping table is constructed.

[0020] In some embodiments of the present disclosure, determining a feature input matrix corresponding to the voxelized point cloud data based on the convolution mapping table includes:

[0021] Based on the convolution mapping table, determining the number of positions of each valid output position;

[0022] Determining the number of rows of the feature input matrix based on the number of positions;

[0023] Determining the number of columns of the feature input matrix based on the number of input channels of the voxelized point cloud data and the size of the convolution kernel;

[0024] Matrix data is filled based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix.

[0025] In some embodiments of the present disclosure, the step of performing matrix data filling based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix includes:

[0026] Obtaining the values ​​of each valid input position on the input channel;

[0027] Determine the filling position of the eigenvalues ​​of each valid input position in the eigenvalue input matrix based on the values ​​on the input channels, the number of rows of the input matrix, and the number of columns of the input matrix;

[0028] According to the filling positions, the point cloud feature values ​​of the effective input positions are filled to obtain the corresponding positions of the feature input matrix to obtain the feature input matrix.

[0029] In some embodiments of the present disclosure, the deforming process of the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix includes:

[0030] Determining the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data;

[0031] Determining the number of matrix columns of the target convolution kernel based on the number of output channels of the convolutional neural network;

[0032] Based on the number of matrix rows of the target convolution kernel and the number of matrix columns of the target convolution kernel, the convolution kernel of the convolutional neural network is deformed into the target convolution kernel.

[0033] A second aspect of the embodiments of the present disclosure provides a device for processing point cloud data, comprising:

[0034] Point cloud data acquisition module, used to acquire voxelized point cloud data;

[0035] A convolution mapping table construction module is used to construct a convolution mapping table based on the voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step size, wherein the index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, an offset position of the convolution kernel, and each valid output position, and each valid input position is a position of each valid point in the voxelized point cloud data;

[0036] A feature input matrix determination module, configured to determine a feature input matrix corresponding to the voxelized point cloud data based on the convolution mapping table, wherein each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to the product of each input channel of the voxelized point cloud data and the number of elements of the convolution kernel;

[0037] A convolution kernel deformation module, used for deforming the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix;

[0038] The sparse convolution operation module is used to perform matrix multiplication calculation on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing based on point cloud data.

[0039] In some embodiments of the present disclosure, the convolution mapping table construction module is used to determine the valid input positions based on the voxelized point cloud data; the convolution mapping table construction module is also used to determine the valid output positions based on the valid input positions, the size of the convolution kernel and the moving step; the convolution mapping table construction module is also used to construct a position index table based on the valid input positions and the valid output positions, wherein the position index table includes the valid input positions and the valid output positions; the convolution mapping table construction module is also used to construct the convolution mapping table based on the position index table and the offset position of the convolution kernel, and generate an index of the convolution mapping table based on the valid output positions.

[0040] In some embodiments of the present disclosure, the convolution mapping table construction module is used to determine the mapping relationship between each valid output position and each valid input position based on the offset position of the convolution kernel; the convolution mapping table construction module is also used to construct the convolution mapping table based on the position index table and the mapping relationship.

[0041] In some embodiments of the present disclosure, the feature input matrix determination module is used to determine the number of positions of each valid output position based on the convolution mapping table; the feature input matrix determination module is also used to determine the number of rows of the feature input matrix based on the number of positions; the feature input matrix determination module is also used to determine the number of columns of the feature input matrix based on the number of input channels of the voxelized point cloud data; the feature input matrix determination module is also used to fill matrix data based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix.

[0042] In some embodiments of the present disclosure, the feature input matrix determination module is used to obtain the numerical value of each valid input position on the input channel; the feature input matrix determination module is also used to determine the filling position of the eigenvalue of each valid input position within the feature input matrix based on the numerical value on the input channel, the number of rows of the input matrix and the number of columns of the input matrix; the feature input matrix determination module is also used to fill the point cloud eigenvalue of each valid input position with the corresponding position of the feature input matrix according to the filling position, so as to obtain the feature input matrix.

[0043] In some embodiments of the present disclosure, the convolution kernel deformation module is used to determine the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data; the convolution kernel deformation module is also used to determine the number of matrix columns of the target convolution kernel based on the number of output channels of the convolution neural network; the convolution kernel deformation module is also used to deform the convolution kernel of the convolution neural network into the target convolution kernel based on the number of matrix rows of the target convolution kernel and the number of matrix columns of the target convolution kernel.

[0044] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0045] a memory for storing a computer program product;

[0046] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in the first aspect above is implemented.

[0047] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0048] The method and device for processing point cloud data of the embodiment of the present disclosure, after obtaining voxelized point cloud data, construct a convolution mapping table for the voxelized point cloud data. The index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, the offset position of the convolution kernel and each valid output position. Based on the convolution mapping table, the feature input matrix corresponding to the voxelized point cloud data is determined, and the convolution kernel of the convolution neural network is deformed according to the feature input matrix to obtain the target convolution kernel, and then by performing matrix multiplication calculation on the feature input matrix and the target convolution kernel, a sparse convolution operation result can be obtained, so that the sparse convolution operation result can be used to perform task processing based on point cloud data. Since the index of the convolution mapping table is determined according to each valid output position, a sparse convolution operation is performed on the voxelized point cloud data based on each valid output position, and the sparse convolution operation result can be obtained by only one matrix multiplication without performing accumulation calculation on the product, and there is no need to retain the intermediate results of the multiplication accumulation in the memory, which greatly reduces the memory space occupied by the intermediate results of the multiplication accumulation during the convolution operation, and facilitates model quantization and integer reasoning during actual deployment. The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0050] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0051] Figure 1 It is a flowchart of a method for processing point cloud data in some embodiments of the present disclosure;

[0052] Figure 2 is a schematic diagram of a convolution mapping table in some examples of the present disclosure;

[0053] Figure 3 is a schematic diagram of a position index table in some embodiments of the present disclosure;

[0054] Figure 4 is a structural block diagram of a device for processing point cloud data in some embodiments of the present disclosure;

[0055] Figure 5 The structure block diagram of an electronic device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0057] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0058] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0059] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0060] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0061] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0062] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0063] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0064] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0065] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0066] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0067] The inventors of the present disclosure have found that the three-dimensional information of an object in a three-dimensional space has a wide range of application scenarios. For example, in the field of autonomous driving or assisted driving, the three-dimensional information of a target object near the vehicle can be obtained in real time through an environmental perception system installed on the vehicle. Using the real-time three-dimensional information of the target object, the position and size of the target object can be determined, and the position and behavior of the target object can also be predicted.

[0068] The three-dimensional structural information of the target object, such as the three-dimensional shape and volume of the target object. The three-dimensional point cloud data of the target object can be obtained by radar scanning, and the three-dimensional contour of the target object is generated from the three-dimensional point cloud data, and then the three-dimensional structural information of the target object is generated according to the three-dimensional contour of the target object. The target image including the target object can also be acquired by an image acquisition device, and the three-dimensional structural information of the target object can be obtained by combining the internal and external parameters of the image acquisition device.

[0069] When the vehicle is in the automatic driving mode or the assisted driving mode, the three-dimensional point cloud data of the objects near the vehicle is usually obtained through the vehicle-mounted laser radar. Among them, the three-dimensional point cloud data of the objects near the vehicle includes the three-dimensional point cloud data of the target object near the vehicle. The target object may include at least one of other vehicles, pedestrians and obstacles near the vehicle. Since the laser radar mainly uses the time of flight (ToF) to calculate the position of the surface point cloud of the target object, the points in the surface point cloud of the target object are unevenly distributed, and some positions on the surface of the target object do not have points in the surface point cloud. Therefore, the three-dimensional point cloud of the target object obtained by the vehicle-mounted laser radar is sparse. In addition, the point cloud features in the three-dimensional point cloud data of the target object are represented by its three-dimensional coordinate values ​​and some accompanying optical information. The order in which the points appear in the three-dimensional point cloud data does not have any effect on the three-dimensional point cloud data. This feature of the point cloud is called disorder.

[0070] Disordered and sparse 3D point cloud data can be densified to obtain voxelized point cloud data. Voxelized point cloud data can include a regular 3D grid structure, such as a cube-shaped 3D grid structure. In the 3D grid structure, the grid positions where radar scanning points exist are valid positions, while the grid positions passed by the laser light during its flight and the grid positions behind the objects that reflect the laser light have no laser reflection information and are invalid positions. The grids at the invalid positions can be filled with zero values ​​after densification.

[0071] In the related art, a deep neural network can be used to process and analyze voxelized point cloud data. The voxelized point cloud data is provided to the deep neural network as input data. All positions in the three-dimensional grid structure of the voxelized point cloud data share a convolution kernel. The convolution kernel traverses the entire input data in a certain order and step size, and performs element-by-element multiplication and accumulation operations with the input data to obtain a sparse convolution operation result. When performing multiplication and accumulation operations on voxelized point cloud data, the convolution kernel used by the deep neural network is also three-dimensional. Each element on the convolution kernel is iteratively multiplied and accumulated with the point cloud features of the input data as the convolution kernel steps. Usually, the valid positions in the three-dimensional grid structure of the voxelized point cloud data are a minority, and the invalid positions filled with zero values ​​are the majority, resulting in the elements of the convolution kernel being multiplied with a large number of zero values. These invalid outputs occupy most of the space and computing power of the convolution operation. The embodiment of the present disclosure is intended to solve this technical problem to at least a certain extent.

[0072] The specific technical solutions of the embodiments of the present disclosure are described in detail below.

[0073] Figure 1 FIG. 1 is a flow chart of a data processing method based on point cloud data in some embodiments of the present disclosure. Figure 1 As shown, the data processing method based on point cloud data includes the following steps:

[0074] S1: Obtain voxelized point cloud data.

[0075] In some examples of the present disclosure, when the vehicle is in the automatic driving mode or the assisted driving mode, the scene around the vehicle can be scanned by the on-board laser radar to obtain the three-dimensional point cloud data of the objects near the vehicle. The on-board image acquisition device can also be used to collect images of the scene near the vehicle including the target object, and the three-dimensional point cloud data of the objects near the vehicle can be obtained by three-dimensionally reconstructing the image. Among them, the three-dimensional point cloud data of the objects near the vehicle includes the three-dimensional point cloud data of the target object. The target object may include other vehicles, pedestrians and animals near the vehicle, etc. After obtaining the above three-dimensional point cloud data, the voxelized point cloud data is obtained by voxelizing the three-dimensional point cloud data.

[0076] In the disclosed embodiment, the grid positions having radar scanning points in the three-dimensional grid structure are defined as valid positions, or the grid positions having points in the three-dimensional point cloud data generated after three-dimensional reconstruction of images in the three-dimensional grid structure are defined as valid positions.

[0077] Define the grid positions that the laser radar passes through during flight and the grid positions behind the objects that reflect the laser as invalid positions, or define the grid positions in the 3D grid structure that do not have points in the 3D point cloud data generated after 3D reconstruction of the image as invalid positions. Fill the invalid positions of the 3D grid structure with zero values.

[0078] The data of each valid position in the three-dimensional grid structure may be N-dimensional data. N may be 8, 16, 32, 64 or other integers greater than 0. The N-dimensional data of each valid position includes feature information and position information corresponding to each valid position. The position information may include the input coordinates of the valid position of the valid pixel point in a preset coordinate system. The preset coordinate system may be a three-dimensional coordinate system with the radar position as the origin of the coordinate system.

[0079] S2: Construct a convolution mapping table based on the voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step.

[0080] The index of the convolution mapping table is generated based on each valid output position. The convolution mapping table includes: the mapping relationship between each valid input position, the offset position of the convolution kernel and each valid output position, and each valid input position is the position of each valid point in the voxelized point cloud data. The offset position of the convolution kernel is the offset position relative to the center of the convolution kernel.

[0081] Each valid position is extracted from the three-dimensional grid structure of the voxelized point cloud data, and each valid position is used as each valid input position of the convolutional neural network.

[0082] Get the size and moving step of the convolutional neural network. The size of the convolutional kernel corresponds to the spatial dimension of the voxelized point cloud data. For example, when the voxelized point cloud data is point cloud data in three-dimensional space, the size of the convolutional kernel can be 3×3. The moving step of the convolutional neural network is the step length of the convolutional kernel moving on the image. The size of the moving step length affects the convolution operation result and the size of the feature map.

[0083] According to the size and moving step of the convolution kernel of the convolutional neural network, a convolution mapping table with indexes generated by each valid output position is constructed. In the convolution mapping table, each valid output position can correspond to at least one valid input position. Through the offset position of the convolution kernel, a mapping relationship is formed between each valid output position and the corresponding valid input position. By querying the convolution mapping table, the valid input position corresponding to each valid output position can be obtained.

[0084] S3: Based on the convolution mapping table, determine the feature input matrix corresponding to the voxelized point cloud data.

[0085] Among them, each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to the product of each input channel of the voxelized point cloud data and the number of elements of the convolution kernel.

[0086] After obtaining the convolution mapping table, the mapping relationship between each valid output position, the offset position of the convolution kernel and each valid input position in the convolution mapping table is used to generate a feature input matrix corresponding to the voxelized point cloud data according to the product of each valid output position and each input channel of the voxelized point cloud data and the number of elements of the convolution kernel.

[0087] S4: Deform the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix.

[0088] The original convolution kernel of the convolutional neural network is deformed. The deformation process is to pull the convolution kernel of the output channel into a column, and finally form a matrix of the target convolution kernel, which has as many columns as the number of convolution kernels.

[0089] S5: Perform matrix multiplication on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing on the target object.

[0090] After the feature input matrix and the target convolution kernel are processed, a matrix multiplication operation is performed on the input matrix and the target convolution kernel to obtain a sparse convolution operation result. The sparse convolution operation result contains the feature processing result of the convolutional neural network on the target object. The target sparse convolution operation result can be input into the detection model to detect the target object, such as position detection or behavior detection of the target object, etc.

[0091] In some examples of the present disclosure, when the target objects include other vehicles and pedestrians near the autonomous driving vehicle, the three-dimensional point cloud data of the environment near the autonomous driving vehicle is collected by the on-board laser radar installed on the autonomous driving vehicle. Among them, the three-dimensional point cloud data includes the three-dimensional point cloud data of other vehicles and pedestrians near the autonomous driving vehicle. The sparse convolution operation results obtained according to the above processing flow S1 to S5 include the feature information of other vehicles and pedestrians near the autonomous driving vehicle. The sparse convolution operation results are input into the pre-trained detection model for vehicle detection and pedestrian detection, and the position and size of other vehicles near the autonomous driving vehicle, as well as the position and size of pedestrians near the autonomous driving vehicle, can be obtained.

[0092] In other examples of the present disclosure, when the target object includes obstacles near the vehicle when it is parked in an underground garage, the three-dimensional point cloud data of the environment near the vehicle is collected by the on-board laser radar installed on the vehicle. Among them, the three-dimensional point cloud data includes the three-dimensional point cloud data of obstacles near the vehicle, and the obstacles near the vehicle may include movable obstacles such as vehicles and pedestrians, and may also include immovable obstacles such as walls and pillars. The sparse convolution operation result obtained according to the above processing flow of S1 to S5 contains the feature information of the obstacles near the vehicle. The sparse convolution operation result is input into the pre-trained detection model for obstacle detection, and the size of the position of the obstacle near the vehicle can be obtained. Combined with the position of the vehicle, the relative position relationship between the vehicle and the nearby obstacles can be obtained. When the relative position relationship between the vehicle and the nearby obstacles meets the preset alarm condition, an alarm is issued to prompt that the vehicle may collide with the nearby obstacles.

[0093] In this embodiment, a laser radar can be used to scan a scene including a target object to obtain three-dimensional point cloud data including point cloud information of the target object, and the three-dimensional point cloud data is densified to obtain voxelized point cloud data. A convolution mapping table indexed by each valid output position is constructed for the voxelized point cloud data. Compared with the method of continuously storing the intermediate results of multiplication and accumulation in the memory when the offset position of the convolution kernel is used as the index, the embodiment of the present disclosure rearranges the original feature input matrix and the original convolution kernel by using the convolution mapping table indexed by each valid output position to obtain the feature input matrix and target convolution kernel required for sparse convolution calculation. Then, according to the rearranged feature input matrix and target convolution kernel, the sparse convolution operation result can be obtained by only one matrix multiplication, without retaining the intermediate results of multiplication and accumulation in the memory, which greatly reduces the memory space occupied during the convolution operation. The sparse convolution operation result can be used to perform tasks such as point cloud classification, object detection, and semantic segmentation on the target object.

[0094] In some embodiments of the present disclosure, S2 may include:

[0095] S2-1: Determine each valid input position based on the voxelized point cloud data.

[0096] Obtain each valid position from the three-dimensional grid structure of the voxelized point cloud data, determine each valid position in the three-dimensional grid structure as each valid input position, and obtain the three-dimensional coordinates of each valid input position in a preset coordinate system. Among them, each valid position in the three-dimensional grid structure is a position filled with non-zero values ​​in the three-dimensional grid structure.

[0097] S2-2: Determine each valid output position based on each valid input position, the size of the convolution kernel and the moving step.

[0098] After obtaining each valid output position, the input coordinates of a valid output position in the preset coordinate system are extracted each time, and the output coordinates corresponding to the currently extracted input coordinates are determined according to the currently extracted input coordinates, the convolution kernel size and the moving step size. The output coordinates are used as the output coordinates of the valid output position, and the corresponding relationship between the input coordinates and the output coordinates is recorded.

[0099] According to the above method, all valid output positions corresponding to all valid input and output positions are obtained, and the corresponding relationship between each valid input position and each valid output position is recorded.

[0100] S2-3: Construct a position index table based on each valid input position and each valid output position.

[0101] A template of the position index table is extracted from the template library, and according to the correspondence between the recorded valid input positions and the valid output positions, each valid input position and each valid output position are filled into the template of the position index table to obtain the position index table. The position index table also includes: an index of each valid input position and an index of each valid output position.

[0102] Figure 3 FIG. 1 is a schematic diagram of a position index table in some embodiments of the present disclosure. Figure 3 As shown, the position index table at least includes: a valid input coordinate hash table generated by the coordinates of each valid input position, and a valid output coordinate hash table generated by the coordinates of each valid output position. In the valid input coordinate hash table, Key represents the index of the valid input coordinate hash table, and Value represents the coordinates of each valid input position in the valid input coordinate hash table. The coordinates of the 0th valid input position in the valid input coordinate hash table are (2,1). After the convolution kernel is moved and the product accumulation is calculated, the corresponding valid output coordinates correspond to the area where A1 is located; the coordinates of the 1st valid input position in the valid input coordinate hash table are (1,2), and the corresponding valid output coordinates correspond to the area where A2 is located. The coordinates of the valid output position corresponding to A1 and the coordinates of the valid output position corresponding to A2 are listed in tables respectively, and the coordinates of the same valid output positions in the two tables are merged to obtain the valid output coordinate hash table. In the valid output coordinate hash table, Key represents the index of the valid output coordinate hash table, and Value represents the output coordinates of the valid output coordinate hash table.

[0103] S2-4: Based on the position index table, construct a convolution mapping table that generates indexes for each valid output position.

[0104] After obtaining the position index table, each valid output position is used as an index to construct a convolution mapping table based on the position index table. Figure 2 Schematic diagram of a convolution mapping table in some examples of the present disclosure. Figure 2 As shown, C in1 , C in2 ,…,C ink Represents channel 1, channel 2, ..., channel k of the voxelized point cloud data. There are K channels inside each channel of the voxelized point cloud data. k Columns correspond to the coordinate offsets of the convolution kernel. The coordinate offsets are the same within each channel, that is, they are all from K0 to K k . Key in the convolution map inIndicates the index of the valid input position. For example, the index of the 0th valid input position is (2,1), the index of the 1st valid input position is (1,2), and so on. The count in the convolution mapping table indicates how many valid input positions each valid output position corresponds to, indicating how many mapping relationships each valid output position corresponds to. Among them, a valid output position, the offset position of the convolution kernel, and a corresponding valid position can generate a mapping relationship. Key in the convolution mapping table out Indices representing valid output positions.

[0105] The feature input matrix is ​​filled in according to the Key out Traversal, Key out Fill in the row with the number corresponding to the key. in Determine which column to fill in by K k For example, in this example, P1 can be filled channel by channel into the 1st column of all channels in the 4th row of the feature input matrix, and P2 can be filled channel by channel into the 0th column of all channels in the 7th row of the feature input matrix.

[0106] In this embodiment, after obtaining each valid input position from the voxelized point cloud data, the valid output positions corresponding to each valid input position can be obtained according to the size of the convolution kernel and the moving step size, and a position index table is constructed according to the mapping relationship between each valid input position and each valid output position. According to the position index table, a convolution mapping table with indexes generated by each valid output position can be quickly constructed, so that the feature input matrix and target convolution kernel required for the sparse convolution operation can be quickly obtained according to the convolution mapping table, so that the sparse convolution operation result can be obtained by only one matrix multiplication for the feature input matrix and target convolution kernel obtained by the rearrangement, without retaining the intermediate results of the multiplication and accumulation in the memory, which helps to significantly reduce the memory space occupied during the convolution operation.

[0107] In some embodiments of the present disclosure, S2-4 may include:

[0108] S2-4-1: Based on the offset position of the convolution kernel, determine the mapping relationship between each valid output position and each valid input position.

[0109] S2-4-2: Based on the position index table and mapping relationship, construct a convolution mapping table that generates indexes for each valid output position.

[0110] In the mapping relationship between each valid output position and each valid input position, each valid output coordinate may correspond to one or more valid input coordinates. Each valid input coordinate will be multiplied by the element on the convolution kernel. Therefore, after obtaining each valid output position and each valid input position in the position index table, a convolution mapping table indexed by each valid output position can be constructed by using the mapping relationship between each valid output position and each valid input position. In the convolution mapping table, the index of each valid output coordinate can be quickly found in the convolution mapping table to find the corresponding valid output coordinate and convolution kernel offset position, and then find one or more corresponding input coordinates and convolution kernel offset positions.

[0111] In this embodiment, the mapping relationship between each valid output position and each valid input position can be obtained by using the convolution kernel offset position. After obtaining each valid output position and each valid input position in the position index table, the mapping relationship can be used to quickly construct a convolution mapping table with indexes generated by each valid output position.

[0112] In some embodiments of the present disclosure, S3 may include:

[0113] S3-1: Based on the convolution mapping table, determine the number of valid output positions in the sparse convolution operation.

[0114] The convolution mapping table can be used to determine how many valid output positions are obtained by the sparse convolution operation.

[0115] S3-2: Determine the number of rows of the feature input matrix based on the number of valid output positions.

[0116] In an example of the present disclosure, when the number of valid output positions is M, the input point cloud features may be organized into a feature input matrix with M rows, wherein each row of the feature input matrix corresponds to an output coordinate, and M is an integer greater than 1.

[0117] S3-3: Determine the number of columns of the feature input matrix based on the number of input channels of the voxelized point cloud data and the size of the convolution kernel.

[0118] The number of columns of the feature input matrix can be segmented according to the input channels of the voxelized point cloud data, and the number of columns of the feature input matrix can be determined according to the number of input channels and the size of the convolution kernel.

[0119] S3-4: Filling matrix data based on the point cloud eigenvalues ​​of each valid input position, the number of rows and columns of the feature input matrix, to obtain the feature input matrix.

[0120] The point cloud eigenvalues ​​at valid positions are divided into two cases.

[0121] The feature source of the first layer of sparse convolution in the neural network: the point cloud is three-dimensionally gridded, all the points in each grid are sent to the same multi-layer perceptron, and the formed features are average pooled to obtain the feature source of the first layer.

[0122] Subsequent feature sources in the neural network: the results obtained after a layer of convolution, normalization, nonlinear activation and other operations on the network.

[0123] By searching the convolution mapping table, multiple sets of mapping relationships can be obtained, each set of mapping relationships includes the coordinates of a valid output position, the convolution kernel offset position, and the coordinates of a valid output position. The eigenvalue of each valid input position on each channel is filled into the corresponding position of the feature input matrix channel by channel.

[0124] In this embodiment, the number of valid output positions in the sparse convolution operation can be determined according to the convolution mapping table, and the number of rows of the feature input matrix can be quickly determined. The number of columns of the feature input matrix can be quickly determined by segmenting the number of input channels of the voxelized point cloud data, and the feature input matrix required for the sparse convolution operation can be quickly generated.

[0125] In some embodiments of the present disclosure, S3-4 may include:

[0126] S3-4-1: Get the value of each valid input position on the input channel.

[0127] After obtaining the voxelized point cloud data, point cloud features are extracted from the voxelized point cloud data, and then the values ​​of the point cloud features on the input channel of the convolutional neural network are obtained.

[0128] S3-4-2: Based on the numerical values ​​on the input channels, the number of rows of the input matrix, and the number of columns of the input matrix, determine the filling position of the eigenvalues ​​of each valid input position in the characteristic input matrix.

[0129] Through the convolution mapping table, we can get the valid input coordinates corresponding to each valid output coordinate, and we can get the filling position of the eigenvalue of each valid input position in the feature input matrix. Figure 2 Key out Fill in the row and column of K. k Determined, the number filled is determined by the corresponding Key in Decision. You can correspond P1 to Key in The eigenvalues ​​of 0 are filled channel by channel into the 1st column of all channels in the 4th row of the feature input matrix, and P2 can be mapped to Key inn The eigenvalues ​​of 1 are filled channel by channel into the 7th row and 0th column of all channels of the feature input matrix.

[0130] S3-4-3: Fill the matrix data based on the eigenvalues ​​and filling positions of each valid input position to obtain a characteristic input matrix.

[0131] In this embodiment, according to the convolution mapping table, the numerical values ​​of the eigenvalues ​​of each valid input position on the input channel can be quickly and accurately filled into the feature input matrix to obtain the feature input matrix.

[0132] In some embodiments of the present disclosure, S4 may include:

[0133] S4-1: Determine the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data.

[0134] S4-2: Determine the number of matrix columns of the target convolution kernel based on the number of output channels of the convolutional neural network.

[0135] S4-3: Based on the number of matrix rows and the number of matrix columns of the target convolution kernel, the convolution kernel is deformed to obtain the target convolution kernel.

[0136] In this embodiment, the original convolution kernel is deformed according to the product of the number of convolution kernel elements and the number of input channels, and the number of output channels, so as to obtain a target convolution kernel suitable for improving the efficiency of sparse convolution operations. This helps to reduce the number of sparse convolution operations through the feature input matrix and the target convolution kernel, saves a lot of storage space, and speeds up the execution efficiency on the AI ​​chip.

[0137] Figure 4 1 is a structural block diagram of a data processing device based on point cloud data in some embodiments of the present disclosure. Figure 4 As shown, a device for processing point cloud data includes:

[0138] The point cloud data acquisition module 100 is used to acquire voxelized point cloud data;

[0139] A convolution mapping table construction module 200 is used to construct a convolution mapping table based on voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step size, wherein the index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, an offset position of the convolution kernel, and each valid output position, and each valid input position is the product of the position of each valid point in the voxelized point cloud data and the number of elements of the convolution kernel;

[0140] A feature input matrix determination module 300 is used to determine a feature input matrix corresponding to the voxelized point cloud data based on the convolution mapping table, wherein each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to each input channel of the voxelized point cloud data;

[0141] A convolution kernel deformation module 400 is used to deform the convolution kernel of the convolutional neural network based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix;

[0142] The sparse convolution operation module 500 is used to perform matrix multiplication calculation on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing based on point cloud data.

[0143] In some embodiments of the present disclosure, a convolution mapping table construction module is used to determine each valid input position based on voxelized point cloud data; the convolution mapping table construction module is also used to determine each valid output position based on each valid input position, the size of the convolution kernel and the moving step; the convolution mapping table construction module is also used to construct a position index table based on each valid input position and each valid output position, wherein the position index table includes each valid input position and the index of each valid input position, and each valid output position and the index of each valid output position; the convolution mapping table construction module is also used to construct a convolution mapping table based on the position index table and the offset position of the convolution kernel.

[0144] In some embodiments of the present disclosure, the convolution mapping table construction module 200 is used to determine the mapping relationship between each valid output position and each valid input position based on the offset position of the convolution kernel; the convolution mapping table construction module 200 is also used to construct a convolution mapping table based on the position index table and the mapping relationship.

[0145] In some embodiments of the present disclosure, the feature input matrix determination module 300 is used to determine the number of positions of each valid output position based on the convolution mapping table; the feature input matrix determination module 300 is also used to determine the number of rows of the feature input matrix based on the number of positions; the feature input matrix determination module 300 is also used to determine the number of columns of the feature input matrix based on the number of input channels of voxelized point cloud data and the number of elements of the convolution kernel; the feature input matrix determination module 300 is also used to fill the matrix data based on the point cloud eigenvalues ​​of each valid input position, the number of rows and columns of the feature input matrix, to obtain the feature input matrix.

[0146] In some embodiments of the present disclosure, the feature input matrix determination module 300 is used to obtain the numerical value of each valid input position on the input channel; the feature input matrix determination module 300 is also used to determine the filling position of the eigenvalue of each valid input position in the feature input matrix based on the numerical value on the input channel, the number of rows of the input matrix and the number of columns of the input matrix; the feature input matrix determination module 300 is also used to fill the point cloud eigenvalue of each valid input position with the corresponding position of the feature input matrix according to the filling position, so as to obtain the feature input matrix.

[0147] In some embodiments of the present disclosure, the convolution kernel deformation module 400 is used to determine the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data; the convolution kernel deformation module 400 is also used to determine the number of matrix columns of the target convolution kernel based on the number of output channels of the convolutional neural network; the convolution kernel deformation module 400 is also used to deform the convolution kernel of the convolutional neural network into the target convolution kernel based on the number of matrix rows and the number of matrix columns of the target convolution kernel.

[0148] It should be noted that the specific implementation of the point cloud data processing device of the embodiment of the present disclosure is similar to the specific implementation of the point cloud data processing method of the embodiment of the present disclosure. Please refer to the description of the point cloud data processing method part for details. In order to reduce redundancy, it will not be repeated.

[0149] In addition, an embodiment of the present disclosure further provides an electronic device, including:

[0150] Memory for storing computer programs;

[0151] The processor is used to execute the computer program stored in the memory, and when the computer program is executed, the method for processing point cloud data described in any of the above embodiments of the present disclosure is implemented.

[0152] Below, reference Figure 5 To describe the electronic device according to the embodiment of the present disclosure. Figure 3 As shown, the electronic device includes one or more processors and memory.

[0153] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0154] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the method for processing point cloud data and / or other desired functions of the various embodiments of the present disclosure described above.

[0155] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0156] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0157] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0158] Of course, to simplify, Figure 5 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0159] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for processing point cloud data according to various embodiments of the present disclosure described in the above part of this specification.

[0160] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0161] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method for processing point cloud data according to various embodiments of the present disclosure described in the above part of this specification.

[0162] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0163] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0164] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0166] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0167] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0168] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0169] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for processing point cloud data, characterized in that: include: Get voxelized point cloud data; Based on the voxelized point cloud data, the size and moving step of the convolution kernel in the convolutional neural network, a convolution mapping table is constructed, wherein the index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, an offset position of the convolution kernel and each valid output position, and each valid input position is a position of each valid point in the voxelized point cloud data; Based on the convolution mapping table, determining a feature input matrix corresponding to the voxelized point cloud data, wherein each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to the product of each input channel of the voxelized point cloud data and the number of elements of the convolution kernel; Based on the feature input matrix, the convolution kernel of the convolutional neural network is deformed to obtain a target convolution kernel corresponding to the feature input matrix; A matrix multiplication calculation is performed on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing based on point cloud data.

2. The method according to claim 1, characterized in that The step of constructing a convolution mapping table based on the voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step size includes: Determining each of the valid input positions based on the voxelized point cloud data; Determine the valid output positions based on the valid input positions, the size of the convolution kernel and the moving step size; Based on the valid input positions and the valid output positions, a position index table is constructed, wherein the position index table includes: the valid input positions and the indexes of the valid input positions, and the valid output positions and the indexes of the valid output positions; The convolution mapping table is constructed based on the position index table and the offset position of the convolution kernel.

3. The method according to claim 2, characterized in that The constructing the convolution mapping table based on the position index table and the offset position of the convolution kernel includes: Determine, based on the offset position of the convolution kernel, a mapping relationship between each valid output position and each valid input position; Based on the position index table and the mapping relationship, the convolution mapping table is constructed.

4. The method according to claim 1, characterized in that The step of determining a feature input matrix corresponding to the voxelized point cloud data based on the convolution mapping table includes: Based on the convolution mapping table, determining the number of positions of each valid output position; Determining the number of rows of the feature input matrix based on the number of positions; Determining the number of columns of the feature input matrix based on the number of input channels of the voxelized point cloud data and the size of the convolution kernel; Matrix data is filled based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix.

5. The method according to claim 4, characterized in that The step of filling matrix data based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix includes: Obtaining the values ​​of each valid input position on the input channel; Determine the filling position of the eigenvalues ​​of each valid input position in the eigenvalue input matrix based on the values ​​on the input channels, the number of rows of the input matrix, and the number of columns of the input matrix; According to the filling positions, the point cloud feature values ​​of the effective input positions are filled to obtain the corresponding positions of the feature input matrix to obtain the feature input matrix.

6. The method according to any one of claims 1 to 5, characterized in that: The deforming the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix includes: Determining the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data; Determining the number of matrix columns of the target convolution kernel based on the number of output channels of the convolutional neural network; Based on the number of matrix rows of the target convolution kernel and the number of matrix columns of the target convolution kernel, the convolution kernel of the convolution neural network is deformed into the target convolution kernel.

7. A device for processing point cloud data, characterized in that: include: Point cloud data acquisition module, used to acquire voxelized point cloud data; A convolution mapping table construction module is used to construct a convolution mapping table based on the voxelized point cloud data, the size of the convolution kernel in the convolutional neural network, and the moving step size, wherein the index of the convolution mapping table is generated based on each valid output position, and the convolution mapping table includes: a mapping relationship between each valid input position, an offset position of the convolution kernel, and each valid output position, and each valid input position is a position of each valid point in the voxelized point cloud data; A feature input matrix determination module, configured to determine a feature input matrix corresponding to the voxelized point cloud data based on the convolution mapping table, wherein each row of the feature input matrix corresponds to each valid output position, and each column of the feature input matrix corresponds to the product of each input channel of the voxelized point cloud data and the number of elements of the convolution kernel; A convolution kernel deformation module, used for deforming the convolution kernel based on the feature input matrix to obtain a target convolution kernel corresponding to the feature input matrix; The sparse convolution operation module is used to perform matrix multiplication calculation on the feature input matrix and the target convolution kernel to obtain a sparse convolution operation result, so as to use the sparse convolution operation result to perform task processing based on point cloud data.

8. The device according to claim 7, characterized in that The convolution mapping table construction module is used to determine the valid input positions based on the voxelized point cloud data; the convolution mapping table construction module is also used to determine the valid output positions based on the valid input positions, the size of the convolution kernel and the moving step; the convolution mapping table construction module is also used to construct a position index table based on the valid input positions and the valid output positions, wherein the position index table includes the indexes of the valid input positions and the valid input positions, and the indexes of the valid output positions and the valid output positions; the convolution mapping table construction module is also used to construct the convolution mapping table based on the position index table and the offset position of the convolution kernel.

9. The device according to claim 8, characterized in that The convolution mapping table construction module is used to determine the mapping relationship between each valid output position and each valid input position based on the offset position of the convolution kernel; the convolution mapping table construction module is also used to construct the convolution mapping table based on the position index table and the mapping relationship.

10. The device according to claim 7, characterized in that The feature input matrix determination module is used to determine the number of positions of each valid output position based on the convolution mapping table; the feature input matrix determination module is also used to determine the number of rows of the feature input matrix based on the number of positions; the feature input matrix determination module is also used to determine the number of columns of the feature input matrix based on the number of input channels of the voxelized point cloud data and the number of elements of the convolution kernel; the feature input matrix determination module is also used to fill the matrix data based on the point cloud feature values ​​of each valid input position and the number of rows and columns of the feature input matrix to obtain the feature input matrix.

11. The device according to claim 10, characterized in that The characteristic input matrix determination module is used to obtain the value of each valid input position on the input channel; the characteristic input matrix determination module is also used to determine the filling position of the characteristic value of each valid input position in the characteristic input matrix based on the value on the input channel, the number of rows of the input matrix and the number of columns of the input matrix; The feature input matrix determination module is further used to fill the point cloud feature values ​​of each valid input position to obtain the corresponding position of the feature input matrix according to the filling position, so as to obtain the feature input matrix.

12. The device according to any one of claims 7 to 11, characterized in that: The convolution kernel deformation module is used to determine the number of matrix rows of the target convolution kernel based on the product of the number of elements of the convolution kernel and the number of input channels of the voxelized point cloud data; the convolution kernel deformation module is also used to determine the number of matrix columns of the target convolution kernel based on the number of output channels of the convolution neural network; the convolution kernel deformation module is also used to deform the convolution kernel of the convolution neural network into the target convolution kernel based on the number of matrix rows of the target convolution kernel and the number of matrix columns of the target convolution kernel.

13. An electronic device, characterized in that: include: A memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 6.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images