A data encoding method for 3D point cloud object detection

Abstract

The present invention discloses a data encoding method for three-dimensional point cloud target detection, comprising: dividing the real space area covered by the three-dimensional point cloud data into a plurality of voxel grids; performing data encoding on the plurality of voxel grids to form a first pseudo image including five channel features, the five channel features including the maximum value of the coordinates of the points in the current voxel grid along the vertical direction, the minimum value of the coordinates along the vertical direction, the maximum reflectivity, the number of points in the voxel grid, and the distribution description coefficient of the point cloud in the voxel grid; splitting the first pseudo image into a second pseudo image including three channel features of the five channel features and a third pseudo image including two channel features of the five channel features; performing feature learning on the second pseudo image and the third pseudo image using a trained convolutional neural network model and obtaining the border and category of the three-dimensional point cloud data. The method of the present invention significantly reduces the amount of data storage and the amount of calculation, and can effectively improve the detection frame rate.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data processing, and in particular relates to a data encoding method for three-dimensional point cloud target detection. Background Art

[0002] 3D point cloud data is a collection of points in a certain coordinate system, which directly provides the 3D spatial information of the target, and its depth information is irreplaceable by other images. This advantage has led to more and more attention paid to 3D target detection based on LiDAR point cloud data.

[0003] In recent years, in the field of 3D object detection based on point cloud data, most researchers have proposed two main methods, including voxel-based methods and point-based methods. Voxel-based methods distribute point clouds to a grid in a spatial coordinate system for 3D convolution, which will generate a large amount of computation; while point-based methods usually involve sampling the farthest point and taking the adjacent points, and cannot effectively parallelize the calculation process, resulting in a large amount of computation and a slow frame rate for 3D object detection. Summary of the invention

[0004] In order to solve the storage disadvantages of existing 3D object detection algorithms and the problem of huge data processing in existing solutions, the present invention provides a data encoding method for 3D point cloud object detection. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] The present invention provides a data encoding method for three-dimensional point cloud target detection, comprising:

[0006] Divide the real space area covered by the three-dimensional point cloud data into multiple voxel grids;

[0007] Data encoding is performed on the plurality of voxel grids to form a first pseudo image including five channel features, wherein the five channel features include a maximum coordinate value of a point in the current voxel grid along the z-axis direction, a minimum coordinate value along the z-axis direction, a maximum reflectivity value, the number of points in the voxel grid, and a distribution description coefficient of a point cloud in the voxel grid;

[0008] Splitting the first pseudo image into a second pseudo image including three channel features of the five channel features and a third pseudo image including two channel features of the five channel features;

[0009] The trained convolutional neural network model is used to perform feature learning on the second pseudo image and the third pseudo image to obtain the border and category of the three-dimensional point cloud data.

[0010] In one embodiment of the present invention, the real space area covered by the three-dimensional point cloud data is divided into a plurality of voxel grids, including:

[0011] Obtain the reflection intensity of each point in the three-dimensional point cloud data and its coordinates in the real coordinate system, where the real coordinate system refers to a three-dimensional coordinate system established with the laser radar as the origin, with the x-axis and y-axis both parallel to the ground and the z-axis perpendicular to the ground;

[0012] The spatial area covered by the three-dimensional point cloud is divided into multiple voxel grids according to a preset voxel size along the plane where the xy axis in the three-dimensional coordinate axis is located, and the number of voxel grids divided on the x-axis and the number of voxel grids divided on the y-axis of the range covered by the point cloud are obtained.

[0013] In one embodiment of the present invention, the distribution description coefficient of the point cloud in the voxel grid is:

[0014]

[0015] Among them, x i and i Respectively represent the x-axis and y-axis coordinate values ​​of the i-th point in the voxel grid, x mean and mean They represent the mean x-axis coordinate value and the mean y-axis coordinate value of all points in the voxel grid respectively.

[0016] In one embodiment of the present invention, the first pseudo image is of [HW 5] scale, wherein H represents the total number of voxel grids into which the real space region is divided along the x-axis direction, W represents the total number of voxel grids into which the real space region is divided along the y-axis direction, and the 5 in [HW 5] represents the five feature channels.

[0017] In one embodiment of the present invention, the second pseudo image is a pseudo image of [HW 3] scale, where 3 in [HW 3] represents three feature channels: the maximum coordinate value along the z-axis direction, the minimum coordinate value along the z-axis direction, and the maximum reflectivity value.

[0018] In one embodiment of the present invention, the third pseudo image is a pseudo image of [HW 2] scale, where 2 in [HW 2] represents two feature channels: the number of points Q in the voxel grid and the distribution description coefficient s of the point cloud in the voxel grid.

[0019] In one embodiment of the present invention, the convolutional neural network model includes a backbone feature learning network, an accompanying auxiliary learning network and a detection head, and using the trained convolutional neural network model to perform feature learning on the second pseudo image and the third pseudo image and obtain the border and category of the three-dimensional point cloud data includes:

[0020] The second pseudo image is input into the backbone feature learning network, and the third pseudo image is input into the accompanying auxiliary learning network. Then, the output features of the accompanying auxiliary learning network are multiplied by the output features of the backbone feature learning network through a Sigmoid function and passed through the detection head to obtain the border and category of the three-dimensional point cloud data.

[0021] In one embodiment of the present invention, the backbone feature learning network is a Unet neural network, and the accompanying auxiliary learning network includes four groups of convolutional layers and activation layers connected in sequence.

[0022] Another aspect of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the data encoding method for three-dimensional point cloud target detection described in any one of the above embodiments.

[0023] Another aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the data encoding method for three-dimensional point cloud target detection as described in any of the above embodiments.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The data encoding method for three-dimensional point cloud target detection of the present invention has a much lower data storage capacity than the existing target detection algorithm, and the data encoding process has a high degree of parallelism, and the amount of calculation is less than the current encoding method. At the same time, the three-dimensional point cloud target detection method of the present invention encodes the voxelized point cloud data, and the encoding result is easier to process in the later network, saving a lot of calculation and effectively improving the detection frame rate.

[0026] 2. The present invention learns features of the second pseudo image and the third pseudo image through two convolutional neural networks respectively, which has higher detection accuracy than learning features through a large network, and the two network output feature layers correspond to the same H and W scales. The channel-by-channel multiplication process can effectively combine local information with overall information, which is beneficial to improving detection accuracy.

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of a data encoding method for three-dimensional point cloud target detection provided by an embodiment of the present invention;

[0029] Figure 2 It is a schematic diagram of a point cloud data voxelization and data encoding process provided by an embodiment of the present invention;

[0030] Figure 3 It is a flowchart of a method for performing target detection using a convolutional neural network model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a data encoding method for three-dimensional point cloud target detection proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0032] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0033] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0034] See also Figure 1 , Figure 1 1 is a flow chart of a data encoding method for three-dimensional point cloud target detection provided by an embodiment of the present invention. The three-dimensional point cloud target detection method comprises:

[0035] S1: Divide the real space area covered by the 3D point cloud data into multiple voxel grids.

[0036] Specifically, if Figure 2 As shown, step S1 of this embodiment includes:

[0037] S11: Obtain the reflection intensity of each point in the three-dimensional point cloud data and its coordinates in the real coordinate system, where the real coordinate system refers to a three-dimensional coordinate system established with the laser radar as the origin, with the x-axis and y-axis both parallel to the ground, and the z-axis perpendicular to the ground.

[0038] Specifically, each point in the point cloud data obtained by the lidar can be expressed as:

[0039] p=(px ,p y ,p z ,p r )

[0040] Among them, p x Indicates the x-axis coordinate of the current point p in the point cloud in the real coordinate system, p y Indicates the y-axis coordinate of the current point in the point cloud in the real coordinate system, p z Indicates the z-axis coordinate of the current point p in the point cloud in the real coordinate system, p r Indicates the reflection intensity of the current point p in the point cloud. The total number of points contained in a frame of point cloud data is N

[0041] It should be noted that the real coordinate system refers to a three-dimensional coordinate system established with the laser radar as the origin, with the x-axis and y-axis both parallel to the ground, and the z-axis perpendicular to the ground.

[0042] Then the collection of point cloud data is expressed as:

[0043] PC={p|0<p x <l,0<p y <w,0<p r <1}

[0044] in, l represents the maximum length range of the point cloud on the x-axis, and w represents the maximum length range of the point cloud on the y-axis, in meters.

[0045] S12: Divide the real space area covered by the three-dimensional point cloud into a plurality of voxel grids according to a preset voxel size along the plane where the xy axis in the three-dimensional coordinate axis is located, and obtain the number of voxel grids divided on the x-axis and the number of voxel grids divided on the y-axis of the range covered by the point cloud, and do not divide on the z-axis. Each voxel grid has the same length and width.

[0046] Specifically, the range covered by the point cloud is divided into the number of voxel grids l on the x-axis. grid And the number of voxel grids divided on the y-axis w grid :

[0047] l grid =l / n l

[0048] w grid =w / n w

[0049] Among them, n l Represents the length of a single voxel grid along the x-axis, n w represents the length of a single voxel grid along the y-axis. In this embodiment, n l =nw Specifically, assuming that the length and width of the preset voxel grid are both 0.16m, and the real space area covered by the 3D point cloud data is [0~61.44m-30.72~30.72m-3~2m], then l grid =l / n l =61.44 / 0.16=384,w grid =w / n w =61.44 / 0.16=384, therefore, it is divided into 384×384 voxel grids on the xy coordinates.

[0050] S2: Data encoding is performed on the multiple voxel grids to form a first pseudo image including five channel features, wherein the five channel features include the maximum coordinate value of a point in the current voxel grid along the z-axis, the minimum coordinate value along the z-axis, the maximum reflectivity, the number of points in the voxel grid, and the distribution description coefficient of the point cloud in the voxel grid.

[0051] Specifically, the points contained in each voxel grid are counted and the five-channel features of the voxel grid are extracted.

[0052] First, the point cloud set P contained in the i-th voxel grid along the x-axis and the j-th voxel grid along the y-axis ij It is expressed as:

[0053] P ij ={p|i×l grid <p x <(i+1)×l grid ,j×w grid <p y <(j+1)×w grid}

[0054] Each point in the 3D point cloud contains 4 channels of information, which are the 3D coordinates of the point in the real world and the reflection intensity. The voxelized point cloud data is encoded, and all the point cloud data contained in a voxel grid are counted and encoded into five channels of information, which include: the maximum value of the coordinate z of the point in the current voxel grid along the z-axis direction; max , the minimum coordinate value z of the point in the current voxel grid along the z-axis min , the maximum reflectivity r of the midpoint of the current voxel grid max , the number of points in the voxel grid Q and the distribution description coefficient s of the point cloud in the grid voxels.

[0055] Specifically, obtain the coordinates of the highest and lowest points of the point cloud on the z-axis in the voxel grid:

[0056] z max =max(Z),Z={p z|p∈P ij}

[0057] z min =min(Z),Z={p z |p∈P ij}

[0058] z max and z min The data encoding of takes into account that the z-axis dimension information is discarded during the voxelization process, while the z-axis contains the height information of the actual object during target detection. As a channel in the original lidar point cloud data, there are different z-axis information between different categories of objects and backgrounds. max and z min This is because the grid points are divided finely enough, which reduces the amount of data while retaining the voxel grid height information, which plays an important role in eliminating interference from road poles, traffic signs, roadside trees, etc. in actual target detection.

[0059] Get the maximum reflectivity of the midpoint of the current voxel grid:

[0060] r max =max(r),r={r z |p∈P ij}

[0061] r max The data encoding takes into account the differences in reflectivity information provided by objects of different distances and materials, which can assist in determining the target voxel grid and eliminate the interference of background point and noise point information.

[0062] Get the number of points Q in the voxel grid:

[0063] Q=card P ij

[0064] The feature extraction of Q provides a density information for the voxel grid and counts the number of voxel grid points in the point cloud.

[0065] Get the distribution description coefficient s of the point cloud in the grid:

[0066]

[0067] Among them, x i and i Respectively represent the x-axis and y-axis coordinate values ​​of the i-th point in the voxel grid, x mean and mean They represent the mean x-axis coordinate value and the mean y-axis coordinate value of all points in the voxel grid respectively.

[0068]

[0069]

[0070] The distribution description coefficient s of the point cloud in the grid represents the average of the Euclidean distances of all points in the voxel grid to their mean point on the xy coordinate plane. This coefficient can effectively describe the distribution law of points in the voxel grid from a bird's eye view, and plays an important role in distinguishing interfering background target voxel grids such as walls.

[0071] like Figure 2 As shown, after the above data encoding, the point cloud data consists of four dimensions (i.e., including the x-axis, the y-axis, the N-axis representing the number of points, and the p-axis representing the number of points). x ,p y ,p z ,p r The four-parameter axis D) is reduced to three dimensions (x-axis, y-axis and axis C representing the five-channel features). The information of the N-axis that originally represents the number of points in the point cloud data is extracted into new channels during the encoding process. Further, after the above data encoding, the three-dimensional point cloud data forms a pseudo image of the scale [HW 5], where H represents the total number of voxel grids along the x-axis direction of the real space area, and W represents the total number of voxel grids along the y-axis direction of the real space area. For example, if the pixels of an image are 720*1280, 720 and 1280 are the H and W of the image, and 5 represents the above five feature channels.

[0072] S3: splitting the first pseudo image into a second pseudo image including three channel features of the five channel features and a third pseudo image including two channel features of the five channel features.

[0073] In this embodiment, the second pseudo image is a pseudo image of [HW 3] scale, where the 3 in [HW 3] represents three feature channels: the maximum coordinate value along the z-axis, the minimum coordinate value along the z-axis, and the maximum reflectivity; the third pseudo image is a pseudo image of [HW 2] scale, where the 2 in [HW 2] represents two feature channels: the number of points Q in the voxel grid and the distribution description coefficient s of the point cloud in the grid voxel.

[0074] S4: Use the trained convolutional neural network model to perform feature learning on the second pseudo image and the third pseudo image, and obtain the border and category of the three-dimensional point cloud data.

[0075] Specifically, see Figure 3 , Figure 3This is a flowchart of using a convolutional neural network model for target detection provided by an embodiment of the present invention. The convolutional neural network model of this embodiment includes a backbone feature learning network, an accompanying auxiliary learning network and a detection head. Specifically, the second pseudo image [HW 3] obtained after data encoding is input into the backbone feature learning network, and the third pseudo image [HW 2] is input into the accompanying auxiliary learning network. Then, the output result of the accompanying auxiliary learning network is multiplied by the output feature of the backbone feature learning network through a Sigmoid function and passed through the detection head to obtain the bounding box and category of the three-dimensional point cloud data. The detection head is composed of several groups of convolutions, which is equivalent to a decoder, and is used to convert the feature layer into the bounding box and category information output by the algorithm. For example, a three-dimensional point cloud represents a car in real space, the bounding box refers to a cube containing the car, and the category represents the type of object, that is, car.

[0076] It should be noted that the backbone feature learning network and the accompanying auxiliary learning network of this embodiment are trained in advance using the current mainstream point cloud target detection task dataset KITTI. In this embodiment, the backbone feature learning network uses the Unet neural network, and the accompanying auxiliary learning network includes 4 groups of convolutional layers and activation layers connected in sequence. The backbone feature learning network outputs a feature layer [HW 64] after feature learning, and the accompanying auxiliary learning network outputs a feature layer [HW 1] with the same length and width as the backbone feature learning network output feature layer, but with 1 channel number.

[0077] The feature learning network of the second pseudo image serves as the main feature learning network of this method. The third pseudo image plays an adjustment role on the main feature layer after feature learning by the accompanying auxiliary learning network. Figure 3 As shown in the figure, after the feature learning of the third pseudo image by the accompanying auxiliary learning network, the nonlinear ability is introduced by a Sigmoid function, and then the obtained feature layer is multiplied by the feature layer of the same scale obtained by the second pseudo image after the backbone feature learning network. The feature layer thus obtained, which has been adjusted by the accompanying auxiliary network, can be used to predict the bounding box and category of the target to be detected by the detection head.

[0078] The method of this embodiment learns features of the second pseudo image and the third pseudo image through two convolutional neural networks respectively, which has higher detection accuracy than learning features through a large network. The reason is that the information in the second pseudo image describes more of the three-dimensional target information corresponding to a single voxel grid, and the number of points in the voxel grid in the third pseudo image and the distribution description coefficient information of the point cloud in the voxel grid describe the distribution position information of the three-dimensional target in the xy plane in the real space. Learning features from both together will confuse local information with overall information. On the contrary, learning features of the third pseudo image through a small network can effectively retain overall information. And the two network output feature layers correspond to the same H and W scales, and the channel-by-channel multiplication process can effectively combine local information with overall information, which is beneficial to improving detection accuracy.

[0079] Furthermore, the effect of the data encoding method of this embodiment can be further illustrated by the following comparative experiment. The method of the embodiment of the present invention and the voxelization method of the existing PointPillars algorithm are used to encode the point cloud data of the same data set, and the voxel grid is set to 0.16m. As shown in Table 1, the amount of data encoded by the method of the embodiment of the present invention is significantly less than the amount of data of the PointPillars voxelization method.

[0080] Table 1 Comparison of the amount of encoded data of different methods

[0081] method Point Pillars voxelization method Method of the present invention Amount of data after encoding 384*384*32*4 384*384*5

[0082] In summary, the data encoding method for three-dimensional point cloud target detection in this embodiment has a much lower data storage volume than the existing target detection algorithm, and the data encoding process has a high degree of parallelism, and the amount of calculation is less than the current encoding method. At the same time, the three-dimensional point cloud target detection method of the present invention encodes the voxelized point cloud data, and the encoding result is easier to process in the later network, saving a lot of calculation and effectively improving the detection frame rate.

[0083] Another embodiment of the present invention provides a storage medium, wherein a computer program is stored in the storage medium, and the computer program is used to execute the steps of the data encoding method for three-dimensional point cloud target detection in the above embodiment. Another aspect of the present invention provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the data encoding method for three-dimensional point cloud target detection as described in the above embodiment are implemented. Specifically, the above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0084] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A data encoding method for three-dimensional point cloud target detection, characterized in that: include: Divide the real space area covered by the three-dimensional point cloud data into multiple voxel grids; Data encoding is performed on the plurality of voxel grids to form a first pseudo image including five channel features, wherein the five channel features include a maximum coordinate value of a point in a current voxel grid along a vertical direction, a minimum coordinate value along a vertical direction, a maximum reflectivity value, the number of points in the voxel grid, and a distribution description coefficient of a point cloud in the voxel grid; Splitting the first pseudo image into a second pseudo image including three channel features of the five channel features and a third pseudo image including two channel features of the five channel features; Using the trained convolutional neural network model to perform feature learning on the second pseudo image and the third pseudo image and obtain the border and category of the three-dimensional point cloud data, The distribution description coefficient of the point cloud in the voxel grid is: Among them, x i and i Respectively represent the x-axis and y-axis coordinate values ​​of the i-th point in the voxel grid, x mean and mean They represent the mean x-axis coordinate value and the mean y-axis coordinate value of all points in the voxel grid, respectively, and Q represents the number of points in the voxel grid; The first pseudo image is of a scale of [HW 5], where H represents the total number of voxel grids divided along the x-axis direction of the real space region, W represents the total number of voxel grids divided along the y-axis direction of the real space region, and 5 in [HW 5] represents the five channel features; The second pseudo image is a pseudo image of [HW 3] scale, where 3 in [HW 3] represents three channel features: the maximum value of the coordinate along the z-axis direction, the minimum value of the coordinate along the z-axis direction, and the maximum value of the reflectivity; The third pseudo image is a pseudo image of [HW 2] scale, where 2 in [HW 2] represents two channel features: the number of points Q in the voxel grid and the distribution description coefficient s of the point cloud in the voxel grid.

2. The data encoding method for three-dimensional point cloud target detection according to claim 1, characterized in that: The real space area covered by the 3D point cloud data is divided into multiple voxel grids, including: Obtain the reflection intensity of each point in the three-dimensional point cloud data and its coordinates in the real coordinate system, where the real coordinate system refers to a three-dimensional coordinate system established with the laser radar as the origin, with the x-axis and y-axis both parallel to the ground and the z-axis perpendicular to the ground; The spatial area covered by the three-dimensional point cloud is divided into multiple voxel grids according to a preset voxel size along the plane where the xy axis in the three-dimensional coordinate system is located, and the number of voxel grids divided on the x-axis and the number of voxel grids divided on the y-axis of the range covered by the point cloud are obtained.

3. The data encoding method for three-dimensional point cloud target detection according to claim 1, characterized in that: The convolutional neural network model includes a backbone feature learning network, an accompanying auxiliary learning network and a detection head, and the trained convolutional neural network model is used to perform feature learning on the second pseudo image and the third pseudo image and obtain the border and category of the three-dimensional point cloud data, including: The second pseudo image is input into the backbone feature learning network, and the third pseudo image is input into the accompanying auxiliary learning network. Then, the output features of the accompanying auxiliary learning network are multiplied by the output features of the backbone feature learning network through a Sigmoid function and passed through the detection head to obtain the border and category of the three-dimensional point cloud data.

4. The data encoding method for three-dimensional point cloud target detection according to claim 3, characterized in that: The backbone feature learning network is a Unet neural network, and the accompanying auxiliary learning network includes four groups of convolutional layers and activation layers connected in sequence.

5. A storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the steps of the data encoding method for three-dimensional point cloud target detection as described in any one of claims 1 to 4.

6. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the data encoding method for three-dimensional point cloud target detection as claimed in any one of claims 1 to 4.

Images