Point cloud data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a method, device, electronic device and storage medium for processing point cloud data, wherein the method comprises: acquiring point cloud data corresponding to a target scene; performing rasterization processing on the acquired point cloud data respectively to obtain point cloud data sets corresponding to each grid; sequentially storing point cloud data sets in the same grid in a storage medium; each of the point cloud data sets contains data of at least one point cloud point; and determining point cloud feature information corresponding to the grid according to the point cloud data set corresponding to each grid. The present disclosure divides point cloud feature extraction into a point cloud rasterization process and a grid feature extraction process, so that the efficiency of point cloud feature extraction is improved by performing overall point cloud feature extraction on the point cloud data set corresponding to each grid in units of grids.

Description

Technical Field

[0001] The present disclosure relates to the technical field of point cloud data processing, and in particular to a point cloud data processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of LiDAR technology, the point cloud data collected by LiDAR includes the accurate location information of the target object. Therefore, the collection of point cloud data using LiDAR is widely used in various fields, such as target detection, 3D target reconstruction, and autonomous driving. Before the above applications are carried out, it is usually necessary to extract features from the collected point cloud data.

[0003] However, since the amount of point cloud data generated by LiDAR is large and updated in real time, if feature extraction is performed directly on the collected point cloud data, the feature extraction speed will be slow. Summary of the invention

[0004] The disclosed embodiments at least provide a point cloud data processing solution, which combines the data reading and writing mechanism to divide the point cloud feature extraction into two stages: rasterization processing and raster feature extraction, thereby improving the efficiency of point cloud feature extraction.

[0005] It mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for processing point cloud data, the method comprising:

[0007] Obtain point cloud data corresponding to the target scene;

[0008] After rasterizing the acquired point cloud data, point cloud data sets corresponding to each grid are obtained; the point cloud data sets in the same grid are sequentially stored in the storage medium; each point cloud data set contains data of at least one point cloud point;

[0009] According to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined.

[0010] By using the above-mentioned point cloud data processing method, the point cloud data corresponding to the target scene can be first obtained, and then the obtained point cloud data can be rasterized respectively, that is, the grid corresponding to the point cloud data is determined, and the point cloud data after rasterization is sequentially stored in the storage medium in units of grids. In this way, before performing feature extraction on each grid, the point cloud data set corresponding to each grid can be read from the storage medium in sequence, and then for each point cloud data set read, the point cloud feature information corresponding to each grid is determined. The above-mentioned extraction method divides the point cloud feature extraction into the point cloud rasterization process and the grid feature extraction process. In this way, by performing overall point cloud feature extraction on the point cloud data set corresponding to each grid in units of grids, the efficiency of point cloud feature extraction is improved.

[0011] In one implementation, the obtained point cloud data is rasterized according to the following steps:

[0012] Based on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, the radar coordinate information of each point cloud point in the point cloud data is converted into corresponding Cartesian coordinate information;

[0013] Each point cloud point is divided into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, wherein the coordinate range of each grid is determined according to the set radar detection range and the preset grid size.

[0014] In one implementation, dividing each point cloud point into a corresponding grid according to the coordinate range of each grid according to the corresponding Cartesian coordinate information includes:

[0015] For each acquired point cloud point, determine the grid to be divided for the point cloud point based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid;

[0016] If the number of divided grids is less than the first preset number, determining whether the number of point cloud points divided into the grid to be divided is less than the second preset number;

[0017] If it is less than the second preset number, the point cloud points are divided into the grid to be divided.

[0018] In one embodiment, determining point cloud feature information corresponding to each grid according to the point cloud data set of the grid includes:

[0019] For each grid point cloud data set, feature enhancement is performed on each point cloud point in the multiple point cloud points based on coordinate information of the multiple point cloud points corresponding to the point cloud data set, and enhanced feature information of each point cloud point is determined; the enhanced feature information has a first feature dimension;

[0020] Determine, based on enhanced feature information of a plurality of point cloud points corresponding to the point cloud data set, coded feature information of the point cloud data set; the coded feature information has a second feature dimension; the number of dimensions of the second feature dimension is greater than the number of dimensions of the first feature dimension;

[0021] Based on the encoded feature information of the point cloud data set, point cloud feature information corresponding to the grid is determined.

[0022] Here, in order to realize grid feature extraction, the embodiment of the present disclosure can first perform feature enhancement on each point cloud point in the multiple point cloud points based on the coordinate information of the multiple point cloud points corresponding to the point cloud data set and obtain the encoded feature information after feature enhancement, which is equivalent to fusing the features of multiple point cloud points in a grid.

[0023] In one embodiment, the step of performing feature enhancement on each of the plurality of point cloud points based on the coordinate information of the plurality of point cloud points corresponding to the point cloud data set to determine the enhanced feature information of each point cloud point includes:

[0024] For each grid point cloud data set obtained, based on the coordinate information of a plurality of point cloud points corresponding to the point cloud data set, determine the center coordinate information corresponding to the point cloud data set;

[0025] For each of the point cloud points, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and the other point cloud points;

[0026] The coordinate information of the point cloud point, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

[0027] In one embodiment, the convolutional neural network includes a convolutional layer and an activation layer; using the convolutional neural network based on enhanced feature information of a plurality of point cloud points corresponding to the point cloud data set to determine the encoded feature information of the point cloud data set includes:

[0028] For each grid point cloud data set obtained, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set is respectively convolved using the convolution kernel of the convolution layer to obtain the convolution feature information corresponding to each point cloud point in the point cloud data set;

[0029] Combining the convolution feature information corresponding to each point cloud point to obtain the convolution feature information corresponding to the point cloud data set;

[0030] The convolution feature information corresponding to the point cloud data set is processed using the activation function of the activation layer to obtain the encoded feature information of the point cloud data set.

[0031] In one implementation, the convolution feature information corresponding to the point cloud data set is processed using the activation function of the activation layer to obtain the encoded feature information of the point cloud data set, including:

[0032] Quantizing the convolution feature information corresponding to the point cloud data set according to a first preset bit width to obtain the convolution feature information after quantization; the first preset bit width is the bit width quantized when the target detection network is deployed to the running platform, and the target detection network is used to detect the target object according to the determined point cloud feature information corresponding to each grid;

[0033] The activation function of the activation layer is used to process the convolution feature information after the quantization process to obtain the encoded feature information of the point cloud data set.

[0034] Here, in order to adapt to the subsequent target detection network, after determining the convolution feature information corresponding to the point cloud data set, the above convolution feature information can be quantized based on the first preset bit width of the target detection network to use the quantized convolution feature information for feature encoding.

[0035] In one implementation, the using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point includes:

[0036] For each of the point cloud points, the coordinate information of the point cloud point, the first deviation information and the second deviation information are quantized according to a second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set;

[0037] The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

[0038] Here, in order to adapt the convolutional neural network used for feature encoding, after determining the enhanced feature information such as the first deviation information and the second deviation information corresponding to the point cloud point, the above-mentioned enhanced feature information can be quantized based on the second preset bit width of the convolutional neural network to determine the quantized enhanced feature information.

[0039] In one embodiment, after determining the point cloud feature information corresponding to each grid, the method further includes:

[0040] The target object is detected based on the determined point cloud feature information corresponding to each grid.

[0041] Here, considering that the point cloud feature information corresponding to the grid presents the spatial distribution of the relevant target objects, the point cloud feature information extracted by the grid feature extraction process can be used as the input information of the target detection network to detect the target object information. Under the premise of fast point cloud feature extraction, the efficiency of target detection is also improved.

[0042] In one embodiment, detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including:

[0043] The point cloud feature information of each grid under the same feature dimension is taken as a group of input information, and different groups of input information corresponding to different feature dimensions are input into the target detection network in parallel to obtain the target object in the target scene.

[0044] In order to better extract the target object information, the point cloud feature information of the grid can be decomposed in the feature dimension before inputting the point cloud feature information into the target detection network. The decomposed point cloud feature information can present the influence of different feature dimensions on the target object detection results, ensuring the detection accuracy.

[0045] In one embodiment, detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including:

[0046] Based on the coordinate information of each point cloud point in the point cloud data set corresponding to each grid, the coordinate information of the target point cloud point representing the grid position is determined;

[0047] The determined point cloud feature information corresponding to the grid is used as the point cloud feature information corresponding to the target point cloud point, and the coordinate information of the target point cloud point and the corresponding point cloud feature information are input into the target detection network to obtain the target object in the target scene.

[0048] Here, considering the correspondence between the grid and the point cloud points, the point cloud feature information of the grid can be mapped to a target point cloud point, and the target point cloud point is used to represent the grid for target object detection, thereby improving the detection efficiency.

[0049] In a second aspect, an embodiment of the present disclosure further provides a radar device, comprising: a radar controller, a radar component, and a field programmable gate array FPGA; wherein the radar controller is electrically connected to the radar component and the FPGA respectively;

[0050] The radar controller is used to control the radar component to scan the target scene to obtain point cloud data corresponding to the target scene;

[0051] The FPGA is used to perform rasterization processing on the point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point; and according to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined; and the target object is detected based on the determined point cloud feature information corresponding to each grid.

[0052] In one embodiment, the FPGA includes an on-chip processor and an on-chip logic circuit;

[0053] The on-chip processor is used to perform rasterization processing on the point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point; and according to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined;

[0054] The on-chip logic circuit is used to detect the target object based on the determined point cloud feature information corresponding to each grid.

[0055] In a third aspect, the present disclosure also provides a point cloud data processing device, the device comprising:

[0056] An acquisition module is used to obtain point cloud data corresponding to the target scene;

[0057] A processing module, used to perform rasterization processing on the acquired point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point;

[0058] The determination module is used to determine the point cloud feature information corresponding to each grid according to the point cloud data set of each grid.

[0059] In a fourth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data processing method as described in the first aspect and any of its various embodiments are performed.

[0060] In a fifth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the point cloud data processing method as described in the first aspect and any of its various embodiments are executed.

[0061] For a description of the effects of the above-mentioned point cloud data processing device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned point cloud data processing method, which will not be repeated here.

[0062] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0064] Figure 1 A flowchart of a method for processing point cloud data provided by the first embodiment of the present disclosure is shown;

[0065] Figure 2 A specific example diagram of the rasterization process in the point cloud data processing method provided in the first embodiment of the present disclosure is shown;

[0066] Figure 3 A specific example diagram of a feature enhancement process in the point cloud data processing method provided in the first embodiment of the present disclosure is shown;

[0067] FIG. 4( a ) shows a specific example diagram before feature encoding in the method for processing point cloud data provided in the first embodiment of the present disclosure;

[0068] FIG4( b ) shows a specific example diagram after feature encoding in the point cloud data processing method provided in the first embodiment of the present disclosure;

[0069] Figure 5 A schematic diagram of a point cloud data processing device provided in the second embodiment of the present disclosure is shown;

[0070] Figure 6 A schematic diagram of an electronic device provided in Embodiment 3 of the present disclosure is shown. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0072] Research has found that in related technologies, since the amount of point cloud data generated by lidar is large and updated in real time, if feature extraction is performed directly on the collected point cloud data, the feature extraction speed will be slow.

[0073] Based on the above research, the present disclosure provides at least one point cloud data processing solution, which combines the data reading and writing mechanism to divide the point cloud feature extraction into two stages: rasterization processing and raster feature extraction, thereby improving the efficiency of point cloud feature extraction.

[0074] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0075] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0076] To facilitate understanding of the present embodiment, a method for processing point cloud data disclosed in the present embodiment is first introduced in detail. The execution subject of the method for processing point cloud data provided in the present embodiment is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing chip, a processing chip such as a microcontroller unit (MCU), a system-on-a-chip (SoC), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a terminal device such as a laser radar device, a user equipment (UE), a mobile device, a user terminal (such as a smart phone), a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method for processing point cloud data can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0077] The point cloud data processing method provided by the embodiment of the present disclosure is described below.

[0078] Embodiment 1

[0079] See also Figure 1 FIG. 1 is a flowchart of a method for processing point cloud data provided by an embodiment of the present disclosure, the method comprising steps S101 to S103, wherein:

[0080] S101, obtaining point cloud data corresponding to the target scene;

[0081] S102, after rasterizing the acquired point cloud data, a point cloud data set corresponding to each grid is obtained; the point cloud data sets in the same grid are sequentially stored in a storage medium; each point cloud data set contains data of at least one point cloud point;

[0082] S103: Determine point cloud feature information corresponding to each grid according to the point cloud data set of the grid.

[0083] Here, in order to facilitate understanding of the processing method of point cloud data provided by the embodiment of the present disclosure, the specific application scenario of the extraction method is first described in detail. The processing method of point cloud data provided by the embodiment of the present disclosure can be mainly applied to target detection, three-dimensional target reconstruction and other fields. Here, target detection is used as an example for illustration. In the related art, in order to determine information such as the position related to the target, after obtaining data information related to the application scenario (such as point cloud data), target detection can be achieved based on a pre-trained target detection network. Here, considering that in the process of relying on the target detection network for target detection, due to the large amount of point cloud data collected by the laser radar and real-time update, this will lead to a slow feature extraction speed, and the slow feature extraction speed directly leads to low efficiency of target detection, especially in applications such as autonomous driving that have high requirements for real-time target detection. The low target detection efficiency directly affects driving safety.

[0084] Precisely to solve the above-mentioned problems, the embodiments of the present disclosure provide a method for processing point cloud data which divides point cloud feature extraction into a point cloud rasterization process and a raster feature extraction process based on different data reading and writing mechanisms. The former process reads the point cloud data for rasterization processing, and writes the data of the rasterized point cloud points sequentially in units of grids. That is, the point cloud data set within the same grid can be written within a continuous address range of the storage medium, thereby facilitating the latter process to sequentially read the point cloud data set corresponding to each grid. The two processes cooperate with each other to ensure the rapid implementation of point cloud feature extraction. In this way, when the above-mentioned extraction method is applied to fields such as target detection, efficient implementation of the application can be ensured.

[0085] The point cloud rasterization process in the disclosed embodiment is a process of rasterizing the point cloud data corresponding to the target scene, and the raster feature extraction process is a process of extracting features from the point cloud data set corresponding to the raster.

[0086] Here, considering that in actual applications, the amount of original point cloud data collected by the laser radar is usually large, it is usually stored in an off-chip memory, such as DDR3\DDR4 in a double data rate synchronous dynamic random access memory (DDR SDRAM). From the perspective of processing efficiency, the point cloud rasterization and grid feature extraction in the point cloud data processing method provided in the embodiment of the present disclosure may rely on the storage of the on-chip memory. Therefore, in the process of rasterization, first, multiple point cloud points corresponding to the point cloud data may be read according to the reading mechanism of the first storage medium (corresponding to the off-chip memory), and then the grid mapped to each point cloud point may be determined. In this way, the point cloud point may be written to the storage address corresponding to the second storage medium (corresponding to the on-chip memory) according to the grid information.

[0087] It should be noted that in the process of writing each point cloud point into the storage medium, the coordinates of each point cloud point falling into a grid can be sorted first, for example, the point cloud points can be sorted in order from small to large in terms of horizontal coordinate values, so that they can be written sequentially according to the sorting results.

[0088] Among them, the process of writing the point cloud data involves out-of-order access to the storage medium. This is mainly because after reading the data of a point cloud point, it is necessary to first determine the grid to which the point cloud point belongs. Here, before writing the point cloud point into the storage medium corresponding to the grid, out-of-order access can be performed on other point cloud points stored in the grid to which the point cloud point belongs. After the sorting operation, the sorted point cloud points can be sequentially written into the storage medium, so as to facilitate the sequential reading of the data in the grid in the subsequent grid feature extraction stage.

[0089] In the embodiment of the present disclosure, the initial feature of a point cloud point may be its coordinate information, which may include a horizontal coordinate value, a vertical coordinate value, a height coordinate value, and a depth coordinate value. The above coordinate values ​​may be represented by {X, Y, Z, I} (e.g. Figure 2 As shown in the example of the point cloud point P1 shown in the figure, it can be known that the feature dimension of the initial feature of a point cloud point is 4-dimensional. For example, when a total of 16 point cloud points are determined in a grid, the sorted 16 point cloud points {P1, P2..., P16} can be sequentially written into the storage address of a grid, such as Figure 2 As shown, information such as the number of point cloud points written into other grids is not specifically exemplified.

[0090] In addition, the process of extracting grid features can be a process of uniformly analyzing the point cloud data set in the grid. In specific applications, the point cloud data set corresponding to each grid can be read first, and then each point cloud point in the point cloud data set in the grid can be feature enhanced. The feature dimension of the point cloud point before feature enhancement is smaller than that of the point cloud point after feature enhancement, which means that the feature information of each point cloud point in the grid is highlighted. Finally, the feature encoding of the corresponding point cloud data set can be realized based on the feature information of each enhanced point cloud point, thereby realizing the feature extraction of the grid.

[0091] It should be noted that, based on the above-mentioned sequential writing operation of the point cloud points, before the raster feature extraction is performed, the relevant data of the written point cloud points can be read sequentially.

[0092] Here, corresponding interfaces can be set for the above-mentioned point cloud rasterization processing operation and raster feature extraction operation respectively. Here, an input interface for the point cloud rasterization operation can be set, and the point cloud data in the storage medium can be randomly accessed by using this input interface. An interface between the point cloud rasterization operation and the raster feature extraction operation can also be set, and this interface can be used to realize sequential access to the rasterized point cloud data stored in the storage medium.

[0093] Among them, the above-mentioned storage medium can be an on-chip memory or an off-chip memory. When the on-chip cache cannot meet the access requirements, the off-chip memory can be used. On a chip with sufficient on-chip cache resources, the on-chip memory can be directly used without reading and writing the off-chip memory, thereby reducing latency and improving real-time access.

[0094] Here, point cloud rasterization and raster feature extraction are key steps of the point cloud data processing method provided by the embodiment of the present disclosure, and they can be explained separately below.

[0095] First, the process of point cloud rasterization can be described in detail.

[0096] The point cloud rasterization process in the embodiment of the present disclosure may rely on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, which may be specifically implemented by the following steps:

[0097] Step 1: Based on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, the radar coordinate information of each point cloud point in the acquired point cloud data is converted into corresponding Cartesian coordinate information;

[0098] Step 2: Divide each point cloud point into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, wherein the coordinate range of each grid is determined according to the set radar detection range and the preset grid size.

[0099] The Cartesian coordinate system here can be established based on the detection plane (such as the ground), and the origin of the established Cartesian coordinate system can coincide with the origin of the radar coordinate system. In this way, when the radar coordinate information of each point cloud point in the point cloud data is projected onto the ground, the Cartesian coordinate information corresponding to the point cloud point can be determined. The Cartesian coordinate information can be the actual distance from the laser radar device when the point cloud point is projected onto the detection plane. After dividing multiple grids based on the set radar detection range and the preset grid size, the grid into which each point cloud point is divided can be determined, thereby realizing point cloud rasterization.

[0100] It should be noted that, when the preset radar detection range is the same, the larger the preset grid size, the fewer the number of divided grids, and the more point cloud points each grid falls into. Similarly, the smaller the preset grid size, the more the number of divided grids, and the fewer the number of point cloud points each grid falls into.

[0101] In the disclosed embodiment, considering that the subsequent grid feature extraction process may be the processing of point cloud data sets in grid units, the number of grids and the number of point cloud points contained in the point cloud data set in the grid will directly affect the speed of grid feature extraction. Here, in order to improve the efficiency of grid feature extraction as much as possible while ensuring the accuracy of feature extraction, the grid can be divided based on the number of grids and the number of point cloud points in the grid. This can be achieved specifically by the following steps:

[0102] Step 1: for each acquired point cloud point, determine the grid to be divided for the point cloud point based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid;

[0103] Step 2: if the number of divided grids is less than the first preset number, determining whether the number of point cloud points divided into the grid to be divided is less than the second preset number;

[0104] Step 3: If the number is less than the second preset number, the point cloud points are divided into grids to be divided.

[0105] Here, in order to facilitate understanding of the above grid division process, an example is given in which the total number of grids is limited to 20 and the number of point cloud points in one grid is limited to 16.

[0106] For a point cloud point obtained, if the grid to be divided for the point cloud point is determined to be the fifth grid based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid, here, if it is determined that the number of divided grids is 4 and has not reached the upper limit of 20, then the fifth grid will be used as the newly divided grid, and the above point cloud point will be used as the first point cloud point in the newly divided grid; if it is determined that the number of divided grids is 5 and has not reached the upper limit of 20, then the fifth grid will be used as the divided grid, and the above point cloud point will be used as the non-first point cloud point in the divided grid, and whether to divide the point cloud point into the fifth grid can be determined based on the judgment result of whether the number of point cloud points divided into the fifth grid is less than or equal to 16. If the number of point cloud points in the fifth grid is already equal to 16, the point cloud point is discarded, otherwise the point cloud point is divided into the fifth grid.

[0107] If for a point cloud point obtained, if based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid, it is determined that the grid to be divided for the point cloud point is the 21st grid, here, if it is determined that the number of divided grids is 20, the upper limit of 20 has been reached, then the point cloud point is discarded and the point cloud rasterization process ends.

[0108] It can be seen that the embodiment of the present disclosure realizes grid division based on the limitation of the number of grids and the number of point cloud points in the grids, thereby ensuring the efficiency of subsequent grid feature extraction.

[0109] Next, the process of grid feature extraction provided by the embodiment of the present disclosure can be described in detail. The grid feature extraction in the embodiment of the present disclosure mainly includes two processes: feature enhancement for point cloud points and feature encoding for grids, which specifically include the following steps:

[0110] Step 1: for each grid point cloud data set, feature enhancement is performed on each point cloud point in the multiple point cloud points based on coordinate information of the multiple point cloud points corresponding to the point cloud data set, and enhanced feature information of each point cloud point is determined; the enhanced feature information has a first feature dimension;

[0111] Step 2: determining the coded feature information of the point cloud data set based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud data set; the coded feature information has a second feature dimension; the number of dimensions of the second feature dimension is greater than the number of dimensions of the first feature dimension;

[0112] Step 3: Based on the encoded feature information of the point cloud dataset, determine the point cloud feature information corresponding to the grid.

[0113] Here, the process of feature enhancement for point cloud points may be a process of amplifying the features of each point cloud point in the point cloud data set of the grid, which may be specifically achieved through the following steps:

[0114] Step 1: for each grid point cloud data set obtained, based on the coordinate information of multiple point cloud points corresponding to the point cloud data set, determine the center coordinate information corresponding to the point cloud data set;

[0115] Step 2: for each point cloud point, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and other point cloud points;

[0116] Step three: Use the coordinate information, the first deviation information and the second deviation information of the point cloud point as enhanced feature information of the point cloud point.

[0117] Here, for each grid point cloud data set read, the center coordinate information corresponding to a point cloud data set can first be determined based on the coordinate information of multiple point cloud points corresponding to the point cloud data set. In a specific application, the coordinate information of each point cloud point can be summed up and then divided by the number of point cloud points contained in the point cloud data set to determine the center coordinate information, which can characterize the centralized distribution trend of each point cloud point in the point cloud data set to a certain extent.

[0118] When determining the center coordinate information corresponding to a grid, the first deviation of any point cloud point falling into the grid from the above center coordinate can be determined, and the second deviation of any point cloud point from other point cloud points in the grid can also be determined.

[0119] Here, for any point cloud point, the coordinate information of the point cloud point can be subtracted from the above-mentioned center coordinate information to determine the first deviation information corresponding to the point cloud point. The first deviation information can characterize the degree of dispersion of data distribution in the point cloud data set to a certain extent. In addition, for any point cloud point, the coordinate information of the point cloud point and the coordinate information of other point cloud points can be subtracted to determine the second deviation information corresponding to the point cloud point. The second deviation information can also characterize the degree of dispersion of data distribution in the point cloud data set to a certain extent.

[0120] In specific applications, in addition to determining the enhanced feature information of the point cloud points based on the first deviation information and the second deviation information mentioned above, other enhanced feature information can also be determined by other processing of the coordinate information of each point cloud point. The embodiments of the present disclosure do not impose specific restrictions on this.

[0121] Here, taking the example that the feature dimension of the initial feature of the point cloud point is 4-dimensional, after feature enhancement, the corresponding enhanced feature information can be 9-dimensional (corresponding to the first feature dimension), that is, the 5-dimensional feature is expanded based on the initial dimension.

[0122] like Figure 3 As shown, a specific example of a 9-dimensional enhanced feature information is obtained by inputting the point cloud points of the 4-dimensional initial feature in a serial data stream manner and then performing feature enhancement. For each point cloud point included in a point cloud data set within a grid, the coordinate information of each point cloud point can be read sequentially. In this way, in the process of feature enhancement, the sequentially read point cloud point information can be subjected to data enhancement processing in a serial data stream manner.

[0123] In order to facilitate data enhancement, when performing feature enhancement for any point cloud point, it can be implemented in combination with the grid index table generated during the rasterization process. Based on the grid index table, the relevant information of other point cloud points belonging to the same grid as the point cloud point can be determined, and then the above-mentioned calculation of the deviation information can be performed to determine the enhanced feature information corresponding to any point cloud point. Under the premise that the point cloud point information is connected to the data enhancement processing operation in the form of a serial data stream, the enhanced feature information output for the point cloud point can also be output in the form of a serial data stream.

[0124] Here, the process of performing feature encoding on the grid may be a process of fusing the features of each point cloud point in the point cloud data set within the grid.

[0125] In the disclosed embodiment, for each grid point cloud data set read, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set can be used as input information of the convolutional neural network, and feature encoding of the point cloud data set can be achieved through convolution operations.

[0126] It should be noted that in order to adapt to the second preset bit width (such as 4 bits) of the above-mentioned convolutional neural network, the coordinate information, first deviation information and second deviation information obtained by the above-mentioned enhancement can be quantized respectively according to the above-mentioned bit width, and the quantized coordinate information, first deviation information and second deviation information can be used as the enhanced feature information after enhancement.

[0127] The convolutional neural network may include a convolutional layer and an activation layer. Feature encoding based on the convolutional neural network may be implemented by the following steps:

[0128] Step 1: for each grid point cloud data set obtained, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set is respectively convolved using the convolution kernel of the convolution layer to obtain the convolution feature information corresponding to each point cloud point in the point cloud data set;

[0129] Step 2: Combine the convolution feature information corresponding to each point cloud point to obtain the convolution feature information corresponding to the point cloud data set;

[0130] Step 3: Use the activation function of the activation layer to process the convolution feature information corresponding to the point cloud dataset to obtain the encoded feature information of the point cloud dataset.

[0131] Here, for a point cloud data set of a grid, the convolution feature information corresponding to each point cloud point in the point cloud data set can be first determined. Here, taking the point cloud data set in a grid including 16 point cloud points, and the dimension of the enhanced feature information of each point cloud point after enhancement is 9 as an example, the convolution operation can be performed on each of the 16 point cloud points, such as using 1x1 convolution to perform convolution operation, so as to obtain the convolution feature information corresponding to each point cloud point.

[0132] By summing up the 16 convolution feature information, the convolution feature information corresponding to the point cloud data set can be obtained. At this time, the point cloud data set can be processed using an activation function to obtain the corresponding encoded feature information, for example, encoded feature information of 32 dimensions (corresponding to the second feature dimension).

[0133] like Figure 4(a) to 4(b) This is a specific example diagram of feature encoding for one of the 32 grids after rasterization. As shown in Figure 4(a), the three axes respectively indicate the number of grids, such as W=32, the number of point cloud points in the grid, such as H=16, and the dimension of the enhanced feature information of each point cloud point after enhancement (such as the enhanced feature information of point cloud point P1), such as C=9. In this way, after encoding according to the above feature encoding method, it can be determined that W=32, H=1, and C=32 indicated by the three axes respectively, that is, the fusion of the features of each point cloud point included in the point cloud data set in a grid is realized, corresponding to the encoded feature information.

[0134] In specific applications, the activation function of ReLU (Rectified Linear Unit) can be used for encoding processing. This function can encode the point cloud data set indicating that the convolution eigenvalue is less than or equal to zero as 0, and encode the point cloud data set indicating that the convolution eigenvalue is greater than zero as 1. The obtained encoded feature information can correspond to the target object at the position of 1, and the background at the position of 0, thereby facilitating subsequent target detection.

[0135] In specific applications, the convolutional neural network for implementing feature encoding can be implemented using a voxel feature encoding (VFE) network. The VFE network can use one-dimensional / streaming sparse convolution technology to implement feature encoding. Since this technology fits the sparse characteristics of the lidar data itself, it can adapt to the data flow density and improve the encoding efficiency, which can further reduce the processing delay and improve the performance.

[0136] It should be noted that in order to adapt to the first preset bit width (such as 8 bits) of the target detection network, the above-mentioned convolution feature information can be quantized according to this bit width, and then the quantized convolution feature information can be encoded to obtain the encoded feature information.

[0137] In the disclosed embodiment, after the point cloud feature information corresponding to each grid is determined according to the above method, the point cloud feature information can be used as input information of the target detection network to detect the target object information.

[0138] The target object determined is different based on different application scenarios. For example, for autonomous driving, the target object may be a person in front of the autonomous driving car or a vehicle in front of the car, and the disclosed embodiment does not impose any specific restrictions on this.

[0139] In specific applications, convolutional neural networks can be used to train target detection networks. When the above-mentioned encoded feature information is input into the convolutional neural network as a sparse graph to participate in the convolution operation, the convolution operation can be performed only on the grids encoded as 1, thereby reducing the amount of convolution operations.

[0140] In the disclosed embodiment, in order to adapt to the data receiving method of the target detection network, feature decomposition can also be performed. Here, the point cloud feature information of each grid under the same feature dimension can be used as a group of input information, and different groups of input information corresponding to different feature dimensions can be input in parallel into the trained target detection network to determine the target object information from the target scene.

[0141] For example, when the dimension of the point cloud feature information of the grid is determined to be 32 dimensions, the point cloud feature information of each grid under the same feature dimension can be used as a group of input information. Different feature dimensions will correspond to different groups of input information, totaling 32 groups of input information. By inputting these 32 groups of input information into the target detection network, the target object information can be determined.

[0142] In the embodiment of the present disclosure, in order to adapt to the data receiving method of the target detection network, the grid scattered point data can also be remapped to the bird's-eye view detection plane, which specifically includes the following steps:

[0143] Step 1: Based on the coordinate information of each point cloud point in the point cloud data set corresponding to each grid, determine the coordinate information of the target point cloud point representing the grid position;

[0144] Step 2: The determined point cloud feature information corresponding to the grid is used as the point cloud feature information corresponding to the target point cloud point, and the coordinate information of the target point cloud point and the corresponding point cloud feature information are input into the target detection network to obtain the target object in the target scene.

[0145] Here, before rasterization, there is a one-to-many relationship between the point cloud points and the grid. After the grid feature extraction, the features of each point cloud point in a grid are fused (corresponding to the feature encoding process). At this time, the grid can be mapped back to the corresponding target point cloud point, and the target object detection can be achieved based on the point cloud feature information corresponding to the coordinate information of the mapped target point cloud point.

[0146] In a specific application, a corresponding interface may be provided between the feature encoding operation and the mapping operation. The encoded feature information is sequentially written into the storage medium. Before the mapping operation, the feature information may be sequentially read from the storage medium.

[0147] Among them, the above-mentioned mapping relationship from grid to point cloud points corresponds to the remapping process of grid scattered point data to the bird's-eye view detection plane. At this time, the coordinate information of the mapped target point cloud points can be marked at the corresponding position of the corresponding bird's-eye view, and the points without grid mapping can be automatically filled with zeros.

[0148] It should be noted that the sparse feature map corresponding to the raster discrete data can be sparsely arranged. Therefore, when the subsequent convolutional neural network can support sparse convolution, the amount of raster feature output data is greatly reduced, which can further reduce the amount of computation.

[0149] In addition, when the laser radar device used in this method is a rotating radar, the rotating radar collects point cloud data at a set angle (such as 15°). Here, the point cloud data collected in one circle (corresponding to 360°) can be used as a frame of point cloud data for point cloud rasterization and raster feature extraction, and the point cloud data collected in half a circle (corresponding to 180°) can be used as a frame of point cloud data for point cloud rasterization and raster feature extraction. The point cloud data collected in a quarter circle (corresponding to 90°) can also be used as a frame of point cloud data for point cloud rasterization and raster feature extraction.

[0150] In order to extract the point cloud feature information corresponding to a complete circle of point cloud data more quickly, two grid feature extraction modules can be used to accelerate the two half-circle point cloud data in parallel, or four grid feature extraction modules can be used to accelerate the four quarter-circle point cloud data in parallel, etc., thereby improving the processing frame rate and further improving the real-time performance of feature extraction.

[0151] Compared with the related art, the outstanding contributions of the point cloud feature extraction method and point cloud data processing method provided by the embodiments of the present disclosure are mainly in the following aspects:

[0152] First, in the process of implementing grid feature extraction, feature enhancement and feature encoding in the embodiment of the present disclosure are sequentially accelerated in the on-chip pipeline, and feature enhancement and feature encoding can be accelerated in parallel;

[0153] Secondly, in the process of realizing point cloud rasterization, the embodiment of the present disclosure uses the point cloud raster index table and the compressed data form of the features for rasterization. In addition, the compressed data form of the encoded features can also be used for feature extraction.

[0154] Based on the point cloud data processing method provided in the above embodiment, the embodiment of the present disclosure further provides a radar device to which the above processing method is applied, the radar device comprising: a radar controller, a radar component and a field programmable gate array FPGA; wherein the radar controller is electrically connected to the radar component and the FPGA respectively;

[0155] A radar controller is used to control the radar component to scan the target scene and obtain point cloud data corresponding to the target scene;

[0156] FPGA is used to perform rasterization processing on the point cloud data respectively to obtain the point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each point cloud data set contains data of at least one point cloud point; and according to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined; and the target object is detected based on the determined point cloud feature information corresponding to each grid.

[0157] Here, the radar component is responsible for driving the laser radar and storing the original point cloud obtained by the laser radar ranging in an off-chip memory such as DDR3\DDR4 or on-chip memory. FPGA obtains the relevant laser radar original point cloud from the on-chip or off-chip memory, calls the PS side of the FPGA heterogeneous system, that is, the ARM side (a microprocessor) core central processing unit (Central Processing Unit, CPU) as a point cloud feature extractor, extracts the features of the laser radar original point cloud, stores the feature information in the on-chip memory, calls the PL side of the FPGA heterogeneous system, that is, the logic circuit, as an artificial neural network accelerator, performs convolution calculations on the feature information, and outputs the detection results to the radar component through the on-chip or off-chip memory. Finally, the radar component is responsible for outputting the detection results to the user end through the network port.

[0158] Wherein, the above-mentioned FPGA includes an on-chip processor and an on-chip logic circuit;

[0159] The on-chip processor is used to perform rasterization processing on the point cloud data respectively, and obtain the point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each point cloud data set contains data of at least one point cloud point; and according to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined;

[0160] The on-chip logic circuit is used to detect the target object based on the determined point cloud feature information corresponding to each grid. The on-chip logic circuit can be used to accelerate the target detection network.

[0161] Here, the point cloud data corresponding to the target scene may be stored in an off-chip memory or an on-chip memory, and the storage medium storing the point cloud data set in the same grid may be an on-chip memory.

[0162] In the disclosed embodiment, a neural network may be used to detect a target object based on point cloud feature information corresponding to each grid.

[0163] The network parameters of the above neural network can be stored in an off-chip memory, and when in use, they are read from the off-chip memory to the on-chip memory, and the on-chip processor and on-chip logic circuit can perform data processing based on the network parameters.

[0164] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0165] Based on the same inventive concept, the embodiment of the present disclosure also provides a point cloud data processing device corresponding to the point cloud data processing method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned point cloud data processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0166] Embodiment 2

[0167] Reference Figure 5 , which is a schematic diagram of the architecture of a point cloud data processing device provided by an embodiment of the present disclosure, the device includes: an acquisition module 501, a processing module 502, and a determination module 503; wherein,

[0168] An acquisition module 501 is used to acquire point cloud data corresponding to a target scene;

[0169] The processing module 502 is used to perform rasterization processing on the acquired point cloud data to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each point cloud data set contains data of at least one point cloud point;

[0170] The determination module 503 is used to determine the point cloud feature information corresponding to each grid according to the point cloud data set of the grid.

[0171] The above-mentioned processing device divides the point cloud feature extraction into a point cloud rasterization process and a raster feature extraction process. In this way, by taking the raster as a unit, the overall point cloud feature extraction is performed on the point cloud data set corresponding to each raster, thereby improving the efficiency of the point cloud feature extraction.

[0172] In one implementation, the processing module 502 is used to perform rasterization processing on the acquired point cloud data according to the following steps:

[0173] Based on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, the radar coordinate information of each point cloud point in the acquired point cloud data is converted into the corresponding Cartesian coordinate information;

[0174] Each point cloud point is divided into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, wherein the coordinate range of each grid is determined according to the set radar detection range and the preset grid size.

[0175] In one implementation, the processing module 502 is used to divide each point cloud point into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid according to the following steps:

[0176] For each acquired point cloud point, determine the grid to be divided for the point cloud point based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid;

[0177] If the number of divided grids is less than the first preset number, determining whether the number of point cloud points divided into the grid to be divided is less than the second preset number;

[0178] If it is less than the second preset number, the point cloud points are divided into grids to be divided.

[0179] In one implementation, the determination module 503 is used to determine the point cloud feature information corresponding to each grid according to the point cloud data set of each grid according to the following steps:

[0180] For each grid point cloud data set, feature enhancement is performed on each point cloud point in the plurality of point cloud points based on coordinate information of the plurality of point cloud points corresponding to the point cloud data set, and enhanced feature information of each point cloud point is determined; the enhanced feature information has a first feature dimension;

[0181] Determine the coded feature information of the point cloud data set based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud data set; the coded feature information has a second feature dimension; the number of dimensions of the second feature dimension is greater than the number of dimensions of the first feature dimension;

[0182] Based on the encoded feature information of the point cloud data set, point cloud feature information corresponding to the grid is determined.

[0183] In one implementation, the determination module 503 is used to perform feature enhancement on each of the multiple point cloud points based on the coordinate information of the multiple point cloud points corresponding to the point cloud data set to determine the enhanced feature information of each point cloud point according to the following steps:

[0184] For each grid point cloud data set obtained, based on the coordinate information of a plurality of point cloud points corresponding to the point cloud data set, determine the center coordinate information corresponding to the point cloud data set;

[0185] For each point cloud point, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and other point cloud points;

[0186] The coordinate information, the first deviation information and the second deviation information of the point cloud point are used as enhanced feature information of the point cloud point.

[0187] In one embodiment, the convolutional neural network includes a convolutional layer and an activation layer; the determination module 503 is used to determine the coded feature information of the point cloud data set based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud data set using the convolutional neural network according to the following steps:

[0188] For each grid point cloud data set obtained, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set is respectively convolved using the convolution kernel of the convolution layer to obtain the convolution feature information corresponding to each point cloud point in the point cloud data set;

[0189] The convolution feature information corresponding to each point cloud point is combined to obtain the convolution feature information corresponding to the point cloud data set;

[0190] The activation function of the activation layer is used to process the convolution feature information corresponding to the point cloud dataset to obtain the encoded feature information of the point cloud dataset.

[0191] In one implementation, the determination module 503 is used to process the convolution feature information corresponding to the point cloud data set using the activation function of the activation layer according to the following steps to obtain the coded feature information of the point cloud data set:

[0192] The convolution feature information corresponding to the point cloud data set is quantized according to a first preset bit width to obtain the convolution feature information after quantization; the first preset bit width is the bit width of quantization when the target detection network is deployed to the running platform, and the target detection network is used to detect the target object according to the determined point cloud feature information corresponding to each grid;

[0193] The activation function of the activation layer is used to process the quantized convolution feature information to obtain the encoded feature information of the point cloud dataset.

[0194] In one implementation, the determination module 503 is configured to use the coordinate information, the first deviation information, and the second deviation information of the point cloud point as enhanced feature information of the point cloud point according to the following steps:

[0195] For each point cloud point, the coordinate information, the first deviation information and the second deviation information of the point cloud point are quantized according to the second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set;

[0196] The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

[0197] In one embodiment, the above device further comprises:

[0198] The detection module 504 is used to detect the target object based on the determined point cloud feature information corresponding to each grid after determining the point cloud feature information corresponding to each grid.

[0199] In one implementation, the detection module 504 is configured to detect the target object based on the determined point cloud feature information corresponding to each grid in accordance with the following steps:

[0200] The point cloud feature information of each grid under the same feature dimension is taken as a group of input information, and different groups of input information corresponding to different feature dimensions are input into the target detection network in parallel to obtain the target object in the target scene.

[0201] In one implementation, the detection module 504 is configured to detect the target object based on the determined point cloud feature information corresponding to each grid in accordance with the following steps:

[0202] Based on the coordinate information of each point cloud point in the point cloud data set corresponding to each grid, the coordinate information of the target point cloud point representing the grid position is determined;

[0203] The determined point cloud feature information corresponding to the grid is used as the point cloud feature information corresponding to the target point cloud point, and the coordinate information of the target point cloud point and the corresponding point cloud feature information are input into the target detection network to obtain the target object in the target scene.

[0204] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0205] Embodiment 3

[0206] The present disclosure also provides an electronic device, such as Figure 6 6, which is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, including: a processor 601, a memory 602, and a bus 603. The memory 602 stores machine-readable instructions (such as Figure 5 In the processing device for point cloud data shown in FIG. 1 , the instructions executed by the acquisition module 501, the processing module 502 and the determination module 503 are correspondingly executed). When the electronic device is running, the processor 601 communicates with the memory 602 through the bus 603. When the machine-readable instructions are executed by the processor 601, the following processing is performed:

[0207] Obtain point cloud data corresponding to the target scene;

[0208] After rasterizing the acquired point cloud data, point cloud data sets corresponding to each grid are obtained; point cloud data sets in the same grid are sequentially stored in the storage medium; each point cloud data set contains data of at least one point cloud point;

[0209] According to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined.

[0210] In one implementation, in the instructions executed by the processor 601, the acquired point cloud data is rasterized according to the following steps:

[0211] Based on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, the radar coordinate information of each point cloud point in the acquired point cloud data is converted into corresponding Cartesian coordinate information;

[0212] Each point cloud point is divided into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, wherein the coordinate range of each grid is determined according to the set radar detection range and the preset grid size.

[0213] In one implementation, the instructions executed by the processor 601 divide each point cloud point into corresponding grids according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, including:

[0214] For each acquired point cloud point, determine the grid to be divided for the point cloud point based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid;

[0215] If the number of divided grids is less than the first preset number, determining whether the number of point cloud points divided into the grid to be divided is less than the second preset number;

[0216] If it is less than the second preset number, the point cloud points are divided into grids to be divided.

[0217] In one implementation, the instructions executed by the processor 601 include determining point cloud feature information corresponding to each grid according to the point cloud data set of each grid, including:

[0218] For each grid point cloud data set, feature enhancement is performed on each point cloud point in the plurality of point cloud points based on coordinate information of the plurality of point cloud points corresponding to the point cloud data set, and enhanced feature information of each point cloud point is determined; the enhanced feature information has a first feature dimension;

[0219] Determine the coded feature information of the point cloud data set based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud data set; the coded feature information has a second feature dimension; the number of dimensions of the second feature dimension is greater than the number of dimensions of the first feature dimension;

[0220] Based on the encoded feature information of the point cloud data set, point cloud feature information corresponding to the grid is determined.

[0221] In one implementation, the instructions executed by the processor 601 include performing feature enhancement on each of the multiple point cloud points based on the coordinate information of the multiple point cloud points corresponding to the point cloud data set to determine the enhanced feature information of each point cloud point, including:

[0222] For each grid point cloud data set obtained, based on the coordinate information of a plurality of point cloud points corresponding to the point cloud data set, determine the center coordinate information corresponding to the point cloud data set;

[0223] For each point cloud point, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and other point cloud points;

[0224] The coordinate information, the first deviation information and the second deviation information of the point cloud point are used as enhanced feature information of the point cloud point.

[0225] In one implementation, in the instructions executed by the processor 601, the convolutional neural network includes a convolutional layer and an activation layer; determining the encoded feature information of the point cloud dataset based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud dataset using the convolutional neural network includes:

[0226] For each grid point cloud data set obtained, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set is respectively convolved using the convolution kernel of the convolution layer to obtain the convolution feature information corresponding to each point cloud point in the point cloud data set;

[0227] The convolution feature information corresponding to each point cloud point is combined to obtain the convolution feature information corresponding to the point cloud data set;

[0228] The activation function of the activation layer is used to process the convolution feature information corresponding to the point cloud dataset to obtain the encoded feature information of the point cloud dataset.

[0229] In one implementation, in the instructions executed by the processor 601, the convolution feature information corresponding to the point cloud data set is processed using the activation function of the activation layer to obtain the encoded feature information of the point cloud data set, including:

[0230] The convolution feature information corresponding to the point cloud data set is quantized according to a first preset bit width to obtain the convolution feature information after quantization; the first preset bit width is the bit width of quantization when the target detection network is deployed to the running platform, and the target detection network is used to detect the target object according to the determined point cloud feature information corresponding to each grid;

[0231] The activation function of the activation layer is used to process the quantized convolution feature information to obtain the encoded feature information of the point cloud dataset.

[0232] In one implementation, in the instructions executed by the processor 601, the coordinate information, the first deviation information, and the second deviation information of the point cloud point are used as enhanced feature information of the point cloud point, including:

[0233] For each point cloud point, the coordinate information, the first deviation information and the second deviation information of the point cloud point are quantized according to the second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set;

[0234] The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

[0235] In one implementation, after determining the point cloud feature information corresponding to each grid, the instructions executed by the processor 601 further include:

[0236] The target object is detected based on the determined point cloud feature information corresponding to each grid.

[0237] In one implementation, in the instructions executed by the processor 601, detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including:

[0238] The point cloud feature information of each grid under the same feature dimension is taken as a group of input information, and different groups of input information corresponding to different feature dimensions are input into the target detection network in parallel to obtain the target object in the target scene.

[0239] In one implementation, in the instructions executed by the processor 601, detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including:

[0240] Based on the coordinate information of each point cloud point in the point cloud data set corresponding to each grid, the coordinate information of the target point cloud point representing the grid position is determined;

[0241] The determined point cloud feature information corresponding to the grid is used as the point cloud feature information corresponding to the target point cloud point, and the coordinate information of the target point cloud point and the corresponding point cloud feature information are input into the target detection network to obtain the target object in the target scene.

[0242] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 601, the steps of the method for processing point cloud data in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0243] The computer program product of the point cloud data processing method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the point cloud data processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0244] The present disclosure also provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0245] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0246] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0247] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0248] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0249] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for processing point cloud data, characterized in that: The method comprises: Obtain point cloud data corresponding to the target scene; The point cloud data obtained are respectively subjected to rasterization processing to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in a storage medium; each point cloud data set contains data of at least one point cloud point; According to the point cloud data set of each grid, determining the point cloud feature information corresponding to the grid; The point cloud feature information is determined by using enhanced feature information obtained by performing feature enhancement on a plurality of point cloud points in the grid; It also includes: performing feature enhancement on multiple point cloud points in the grid in the following manner: For a point cloud data set formed by a plurality of point cloud points in the grid, based on coordinate information of the plurality of point cloud points in the point cloud data set, determining center coordinate information corresponding to the point cloud data set; For each of the point cloud points, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and the other point cloud points; Using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point; The using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point includes: For each of the point cloud points, the coordinate information of the point cloud point, the first deviation information and the second deviation information are quantized according to a second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set; The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

2. The processing method according to claim 1, characterized in that: The point cloud data is rasterized according to the following steps: Based on the conversion relationship between the radar coordinate system and the Cartesian coordinate system, the radar coordinate information of each point cloud point in the point cloud data is converted into corresponding Cartesian coordinate information; Each point cloud point is divided into a corresponding grid according to the corresponding Cartesian coordinate information according to the coordinate range of each grid, wherein the coordinate range of each grid is determined according to the set radar detection range and the preset grid size.

3. The processing method according to claim 2, characterized in that: The step of dividing each point cloud point into corresponding grids according to the corresponding Cartesian coordinate information according to the coordinate range of each grid includes: For each acquired point cloud point, determine the grid to be divided for the point cloud point based on the Cartesian coordinate information corresponding to the point cloud point and the coordinate range of each grid; If the number of divided grids is less than the first preset number, determining whether the number of point cloud points divided into the grid to be divided is less than the second preset number; If it is less than the second preset number, the point cloud points are divided into the grid to be divided.

4. The processing method according to any one of claims 1 to 3, characterized in that: According to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined, including: For each grid point cloud data set, feature enhancement is performed on each point cloud point in the multiple point cloud points based on coordinate information of the multiple point cloud points corresponding to the point cloud data set, and enhanced feature information of each point cloud point is determined; the enhanced feature information has a first feature dimension; Determine, based on enhanced feature information of a plurality of point cloud points corresponding to the point cloud data set, coded feature information of the point cloud data set; the coded feature information has a second feature dimension; the number of dimensions of the second feature dimension is greater than the number of dimensions of the first feature dimension; Based on the encoded feature information of the point cloud data set, point cloud feature information corresponding to the grid is determined.

5. The processing method according to claim 4, characterized in that: The convolutional neural network includes a convolutional layer and an activation layer; using the convolutional neural network to determine the coded feature information of the point cloud data set based on the enhanced feature information of the plurality of point cloud points corresponding to the point cloud data set, including: For each grid point cloud data set obtained, the enhanced feature information of multiple point cloud points corresponding to the point cloud data set is respectively convolved using the convolution kernel of the convolution layer to obtain the convolution feature information corresponding to each point cloud point in the point cloud data set; Combining the convolution feature information corresponding to each point cloud point to obtain the convolution feature information corresponding to the point cloud data set; The convolution feature information corresponding to the point cloud data set is processed using the activation function of the activation layer to obtain the encoded feature information of the point cloud data set.

6. The processing method according to claim 5, characterized in that: The convolution feature information corresponding to the point cloud data set is processed using the activation function of the activation layer to obtain the encoded feature information of the point cloud data set, including: Quantizing the convolution feature information corresponding to the point cloud data set according to a first preset bit width to obtain the convolution feature information after quantization; the first preset bit width is the bit width quantized when the target detection network is deployed to the running platform, and the target detection network is used to detect the target object according to the determined point cloud feature information corresponding to each grid; The activation function of the activation layer is used to process the convolution feature information after the quantization process to obtain the encoded feature information of the point cloud data set.

7. The processing method according to any one of claims 1 to 3, characterized in that: After determining the point cloud feature information corresponding to each grid, it also includes: The target object is detected based on the determined point cloud feature information corresponding to each grid.

8. The processing method according to claim 7, characterized in that: Detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including: The point cloud feature information of each grid under the same feature dimension is taken as a group of input information, and different groups of input information corresponding to different feature dimensions are input into the target detection network in parallel to obtain the target object in the target scene.

9. The processing method according to claim 7, characterized in that: Detecting the target object based on the determined point cloud feature information corresponding to each grid is performed by a target detection network, including: Based on the coordinate information of each point cloud point in the point cloud data set corresponding to each grid, the coordinate information of the target point cloud point representing the grid position is determined; The determined point cloud feature information corresponding to the grid is used as the point cloud feature information corresponding to the target point cloud point, and the coordinate information of the target point cloud point and the corresponding point cloud feature information are input into the target detection network to obtain the target object in the target scene.

10. A radar device, characterized in that: include: A radar controller, a radar component and a field programmable gate array FPGA; wherein the radar controller is electrically connected to the radar component and the FPGA respectively; The radar controller is used to control the radar component to scan the target scene to obtain point cloud data corresponding to the target scene; The FPGA is used to perform rasterization processing on the point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point; and according to the point cloud data set of each grid, point cloud feature information corresponding to the grid is determined; and the target object is detected based on the determined point cloud feature information corresponding to each grid; The point cloud feature information is determined by using enhanced feature information obtained by performing feature enhancement on a plurality of point cloud points in the grid; The FPGA is also used to perform feature enhancement on multiple point cloud points in the grid in the following manner: For a point cloud data set formed by a plurality of point cloud points in the grid, based on coordinate information of the plurality of point cloud points in the point cloud data set, determining center coordinate information corresponding to the point cloud data set; For each of the point cloud points, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and the other point cloud points; Using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point; The using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point includes: For each of the point cloud points, the coordinate information of the point cloud point, the first deviation information and the second deviation information are quantized according to a second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set; The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

11. The radar device according to claim 10, characterized in that: The FPGA includes an on-chip processor and an on-chip logic circuit; The on-chip processor is used to perform rasterization processing on the point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point; And according to the point cloud data set of each grid, the point cloud feature information corresponding to the grid is determined; The on-chip logic circuit is used to detect the target object based on the determined point cloud feature information corresponding to each grid.

12. A point cloud data processing device, characterized in that: The device comprises: An acquisition module is used to obtain point cloud data corresponding to the target scene; A processing module, used to perform rasterization processing on the acquired point cloud data respectively to obtain point cloud data sets corresponding to each grid; the point cloud data sets in the same grid are sequentially stored in the storage medium; each of the point cloud data sets contains data of at least one point cloud point; A determination module, used to determine point cloud feature information corresponding to each grid according to the point cloud data set of the grid; The point cloud feature information is determined by using enhanced feature information obtained by performing feature enhancement on a plurality of point cloud points in the grid; It also includes: performing feature enhancement on multiple point cloud points in the grid in the following manner: For a point cloud data set formed by a plurality of point cloud points in the grid, based on coordinate information of the plurality of point cloud points in the point cloud data set, determining center coordinate information corresponding to the point cloud data set; For each of the point cloud points, based on the coordinate information of the point cloud point and the determined center coordinate information, determine the first deviation information corresponding to the point cloud point; and based on the coordinate information of the point cloud point and the coordinate information of other point cloud points in the point cloud data set except the point cloud point, determine the second deviation information between the point cloud point and the other point cloud points; Using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point; The using the coordinate information of the point cloud point, the first deviation information, and the second deviation information as enhanced feature information of the point cloud point includes: For each of the point cloud points, the coordinate information of the point cloud point, the first deviation information and the second deviation information are quantized according to a second preset bit width to obtain the coordinate information, the first deviation information and the second deviation information after quantization; the second preset bit width is the bit width of quantization when the convolutional neural network is deployed to the running platform; the convolutional neural network determines the encoding feature information of the point cloud data set based on the enhanced feature information of the multiple point cloud points corresponding to the point cloud data set; The quantized coordinate information, the first deviation information and the second deviation information are used as enhanced feature information of the point cloud point.

13. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data processing method as described in any one of claims 1 to 9 are performed.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the point cloud data processing method according to any one of claims 1 to 9 are executed.

Images