Point cloud automatic labeling method and device based on neural network, and medium

Abstract

The invention discloses an automatic point cloud labeling method and device based on a neural network and a medium, and relates to the field of data processing, and the method comprises the steps: obtaining an original point cloud sequence, and dividing the point cloud sequence into a plurality of subsequences; converting each point in each frame of point cloud data from a world coordinate system to a relative coordinate system to obtain a relative coordinate system value of each point; uniformly dividing each frame of point cloud data into a plurality of preset voxel blocks, and coding the voxel blocks according to whether the voxel blocks contain valid points or not to obtain valid voxels and empty voxels contained in each frame of point cloud data; obtaining a spatial feature vector of each frame of point cloud data; extracting time sequence feature vectors of the subsequences by using a neural network; performing feature fusion to obtain a fused feature vector, and generating an initial detection frame; and adjusting the size of the initial detection frame according to the tracking trajectory to obtain a target detection frame. The spatial features and the time features of the point cloud can be fully utilized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device and medium for automatic annotation of point clouds based on a neural network. Background Art

[0002] The annotation of 3D point cloud data plays a vital role in the current deep learning system, especially in the research and development process of systems such as robots, embodied functions, and autonomous driving. At the same time, higher and higher requirements are put forward on the data closed loop and the efficiency and quality of data annotation on the data side. For 3D point cloud data, the current method of data annotation that relies solely on annotators generally has problems with annotation efficiency, annotation precision, and relatively difficult size estimation. Under the circumstance that the method of data annotation that relies solely on annotators is increasingly unable to meet the high-precision and high-efficiency annotation requirements proposed by high-precision algorithms, the automatic annotation method is used to annotate the data, and then the annotators perform attribute supplementation and a small amount of fine-tuning, omission correction, and error filtering. This has gradually become an important means to improve the efficiency of point cloud annotation and the quality of annotation results.

[0003] However, some of the current mainstream automatic labeling methods do not fully utilize the temporal characteristics of point clouds. For example, some labeling methods use a single frame for detection output, while some labeling methods directly splice adjacent frames. These labeling methods will result in detection accuracy and recall rates that are not ideal, and there is also the problem of inaccurate detection frame size.

[0004] Therefore, it is necessary to provide a method for data annotation for three-dimensional point cloud data. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method, device and medium for automatic annotation of point clouds based on a neural network, so as to solve the problem that the current automatic annotation methods do not fully utilize the temporal characteristics of point clouds.

[0006] According to a first aspect, an embodiment of the present invention provides a method for automatic annotation of point clouds based on a neural network, the method comprising: Obtain the original point cloud sequence, and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; Each frame of point cloud data after conversion is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks without valid points are empty voxels; Extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence based on the spatial feature vector of each frame of point cloud data; Extract the temporal feature vector of the subsequence using neural network; Perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; According to the spatial relationship between the initial detection frames of the subsequences, the initial detection frames between the subsequences are continuously tracked to obtain the tracking trajectory; The size of the initial detection frame is adjusted according to the tracking trajectory to obtain the target detection frame.

[0007] In combination with the first aspect, in a first implementation of the first aspect, converting each point in each frame of point cloud data from the world coordinate system to the relative coordinate system according to the pose of each frame of point cloud data in the subsequence in the world coordinate system to obtain the relative coordinate system value of each point specifically includes: Determine the pose of each frame of point cloud data in the subsequence relative to the world coordinate system, and convert each point under each point cloud data in the subsequence into homogeneous coordinates; According to the homogeneous coordinates corresponding to each point, each point in the subsequence is converted from the world coordinate system to the relative coordinate system; According to the point cloud data corresponding to each point and the position of the first frame of the subsequence point cloud data relative to the world coordinate system, the coordinate values ​​in the aligned secondary coordinates are converted to determine the coordinate value of each point in the relative coordinate system; According to the preset maximum and minimum values ​​of the point cloud intensity, the intensity values ​​in the secondary coordinates are normalized to determine the intensity value of each point in the relative coordinate system.

[0008] In combination with the first aspect, in the second implementation of the first aspect, the converted each frame of point cloud data is evenly divided into a plurality of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data, specifically including: Determine the effective area to be processed for each frame of point cloud data in each subsequence, and evenly divide the effective area into a number of preset voxel blocks, and determine the points under the point cloud data contained in each voxel block; the preset voxel block is set to a size of Cube space, represents the size of the voxel block in the x-axis direction, represents the size of the voxel block in the y-axis direction, Indicates the size of the voxel block in the z-axis direction; Each voxel block is voxel-encoded using a preset encoder to determine valid voxels in the voxel block; the input data of the encoder is the relative coordinate system values ​​of all points in the voxel block.

[0009] In combination with the second implementation of the first aspect, in the third implementation of the first aspect, the extracting the time series feature vector of the subsequence by using a neural network specifically includes: The spatial feature vector corresponding to each frame of point cloud data is evenly divided into several sub-partitions, and the voxel blocks contained in each sub-partition are determined; Concatenate the feature vectors of all voxel blocks in the subpartition forward and backward to obtain the subspace feature vector of the subpartition; The subspace feature vectors of all subpartitions of the subsequence are collected, and the time series features of all subspace feature vectors contained in the subsequence are extracted through a neural network to obtain a time series feature vector.

[0010] In combination with the first aspect, in a fourth implementation of the first aspect, the feature fusion method is: Obtain the point spatial feature vector of each point in the spatial feature vector and the point temporal feature vector of each point in the temporal feature vector, and concatenate the point spatial feature vector and the point temporal feature vector to obtain a point fusion vector; The point fusion vectors of each point in the subsequence are collected to obtain the feature fusion vector.

[0011] In combination with the first aspect, in a fifth implementation of the first aspect, converting empty voxels into dense voxel features is achieved by assigning a value of 0 to all empty voxels.

[0012] In combination with the first aspect, in a sixth implementation of the first aspect, adjusting the size of the initial detection frame according to the tracking trajectory to obtain the target detection frame specifically includes: Using each initial detection frame in the tracking trajectory, a tracking point cloud at a corresponding position is cut out from the corresponding frame in the subsequence; Convert the coordinates of the tracking point cloud to the detection frame coordinate system with the initial detection frame as the origin to obtain the converted point cloud; Superimpose the transformed point clouds of all initial detection boxes in the trajectory to form an adjusted point cloud; Input the adjusted point cloud into the point cloud PointNet neural network for regression processing, and the target size is obtained by the output of the point cloud PointNet neural network; Reassign the target size to all initial detection boxes in the tracking trajectory to obtain the target detection box. According to the second aspect, an embodiment of the present invention further provides a point cloud automatic annotation device based on a neural network, the device comprising: The sequence division module is used to obtain the original point cloud sequence and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; The coordinate conversion module is used to convert each point in each frame of point cloud data from the world coordinate system to the relative coordinate system according to the position of each frame of point cloud data in the subsequence in the world coordinate system, and obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; The voxel encoding module is used to evenly divide each frame of point cloud data after conversion into a number of preset voxel blocks, and encode the voxel blocks according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks not containing valid points are empty voxels; The spatial feature module is used to extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence according to the spatial feature vector of each frame of point cloud data; A time series feature module is used to extract the time series feature vector of the subsequence using a neural network; A feature fusion module is used to fuse the spatial feature vectors and the temporal feature vectors of the subsequences to obtain a fused feature vector, perform a convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; A trajectory tracking module is used to continuously track the initial detection frames between the subsequences according to the spatial relationship between the initial detection frames between the subsequences to obtain a tracking trajectory; The size optimization module is used to adjust the size of the initial detection frame according to the tracking trajectory to obtain the target detection frame.

[0013] According to the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned neural network-based point cloud automatic labeling methods are implemented.

[0014] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described neural network-based point cloud automatic labeling methods.

[0015] The neural network-based point cloud automatic annotation method, device and medium of the present invention can make full use of the spatial characteristics and temporal characteristics of the point cloud. The accuracy and recall rate of the detection results generated based on the fusion of these two features are significantly improved. By optimizing the size of the original point cloud data based on the tracking results, a more accurate detection frame size can be obtained. Since the local features of the detected object have temporal continuity, when extracting the temporal features, the method of first dividing the subspace and then extracting is adopted, rather than calculating and extracting in the entire space, which can greatly reduce the complexity of the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings: Figure 1 A schematic diagram of the process of the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 2 A schematic diagram of a subsequence after voxel encoding is completed in the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 3 A schematic diagram showing the extraction of temporal features of subsequences in the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 4 A schematic diagram showing sub-partitioning and tile mapping in the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 5 A schematic diagram of a subspace feature vector of a subsequence in the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 6 A schematic diagram showing the extraction of temporal features by the point cloud automatic annotation method based on a neural network provided by the present invention; Figure 7 A schematic diagram of the structure of the point cloud automatic annotation method based on a neural network provided by the present invention is shown; Figure 8 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] As the iterative cycle of AI product development represented by robots and autonomous driving continues to increase, higher and higher requirements are currently being placed on the data side for data closure and the efficiency and quality of data annotation. Among them, the annotation of three-dimensional point cloud data plays a vital role in the current deep learning system, especially in the research and development process of systems such as robots, embodied functions, and autonomous driving. For example, in robots and autonomous driving systems, lidar is an important sensor that is often used to achieve environmental perception, dynamic and static obstacle recognition, map matching, positioning, construction, and other functions. The realization of these functions is highly dependent on various high-precision and high-performance point cloud algorithms. Training these algorithms requires a large amount of labeled lidar point cloud data; in the pure visual autonomous driving solution based on Bird's Eye View (BEV) + Transformer, it is also necessary to train the algorithm in the true value of three-dimensional space. Three-dimensional point cloud data is an excellent carrier for providing three-dimensional space expression. Therefore, efficient and high-quality point cloud data annotation has become an important link to improve the iteration speed of algorithms and products, optimize algorithm training effects, and improve the upper limit of algorithm capabilities.

[0019] For three-dimensional point cloud data, the current method of relying solely on labeling personnel to label data generally has problems with labeling efficiency, labeling precision and relatively difficult size estimation. This is due to the large subjectivity of human beings. It is difficult to achieve consistent labeling of the same object in the time dimension, and the trajectory labeling of the object to be labeled is prone to jitter, deformation and other operations.

[0020] Therefore, when the method of relying solely on labelers for data labeling is increasingly unable to meet the high-precision and high-efficiency labeling requirements proposed by high-precision algorithms, labeling the data to be labeled using automatic labeling methods, and then having labelers perform attribute supplements and a small amount of fine-tuning, omission correction, error filtering and other operations has gradually become an important means to improve the efficiency of point cloud labeling and the quality of labeling results.

[0021] However, some of the current mainstream automatic labeling methods do not fully utilize the temporal characteristics of point clouds. For example, some labeling methods use a single frame for detection output, while some labeling methods directly splice adjacent frames. These labeling methods will result in detection accuracy and recall rates that are not ideal, and there is also the problem of inaccurate detection frame size.

[0022] In summary, it is necessary to provide a method for data annotation for three-dimensional point cloud data.

[0023] In order to solve the above problems, a point cloud automatic annotation method based on a neural network is provided in this embodiment, aiming to better annotate three-dimensional point cloud data. The point cloud automatic annotation method based on a neural network in the embodiment of the present invention can be used in electronic devices, including but not limited to computers, mobile terminals, etc. Figure 1 is a flow chart of a method for automatic annotation of point clouds based on a neural network according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: S10. Obtain an original point cloud sequence, and divide the point cloud sequence into a plurality of subsequences, wherein each subsequence contains point cloud data of a plurality of consecutive frames.

[0024] First, get the original point cloud continuous sequence , They are the original point cloud data collected at N consecutive moments from 1 to N. Afterwards, the original point cloud continuous sequence is divided into several subsequences, each of which contains at most M consecutive frames of point cloud data, that is, M frames of point cloud data at consecutive moments.

[0025] In this embodiment, the subsequences are divided in such a way that M continuous frames of data are extracted from the original point cloud continuous sequence in order of time to obtain a subsequence. If enough M frames of point cloud data cannot be extracted in the last extraction, the actual number of remaining frames is used as the basis for extraction, thereby obtaining M subsequences.

[0026] For any subsequence ,in For the subsequence Frame, it can be seen The CCP has Frame point cloud data, For the subsequence Frame point cloud data.

[0027] S20. According to the pose of each frame of point cloud data in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point, wherein the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system.

[0028] In this embodiment, each frame of point cloud data in the acquired original point cloud continuous sequence corresponds to the position and posture of the frame of point cloud data relative to the world coordinate system. , and, posture is a 4×4 coordinate transformation matrix. Then for the subsequences obtained by division In this step, you need to put The point cloud coordinate system is transferred to the origin set as The relative coordinate system is used, and the intensity normalization operation is performed later.

[0029] Specifically, assuming For the The first Frame point cloud data, its position in the world coordinate system is ,for Any point in the point cloud data , it is necessary to set this point based on the pose Translate to Origin Set to (No. The relative coordinate value of the relative coordinate system of the first frame of point cloud data in the subsequence) ,in, Indicate point The coordinate in the x-axis direction in the relative coordinate system, Indicate point The coordinate in the y-axis direction in the relative coordinate system, Indicate point The coordinate in the z-axis direction in the relative coordinate system, Indicate point Intensity value in relative coordinate system (scale factor).

[0030] In this embodiment, step S20 specifically includes: S21, determining the position and posture of each frame of point cloud data in the subsequence relative to the world coordinate system, and converting each point under each point cloud data in the subsequence into homogeneous coordinates.

[0031] First, Convert to homogeneous coordinates ,in, Indicate point The coordinate in the x-axis direction in the world coordinate system, Indicate point The coordinate in the y-axis direction in the world coordinate system, Indicate point The coordinate in the z-axis direction in the world coordinate system, Indicate point The intensity value (scale factor) in world coordinates.

[0032] S22. According to the homogeneous coordinates corresponding to each point, each point in the subsequence is converted from the world coordinate system to the relative coordinate system.

[0033] S23, according to the point cloud data corresponding to each point and the position of the first frame of point cloud data of the subsequence relative to the world coordinate system, the coordinate values ​​in the aligned secondary coordinates are converted into a coordinate system to determine the coordinate value of each point in the relative coordinate system.

[0034] Next, align the secondary coordinates to perform coordinate transformation and convert the points The world coordinate system is transformed to the origin and is set to The relative coordinate system of the coordinate system is transformed into The homogeneous coordinates of are ,at this time, The coordinate value of The x, y, and z coordinate values ​​of the

[0035] S24, normalize the intensity values ​​in the aligned secondary coordinates according to the preset maximum and minimum values ​​of the point cloud intensity, and determine the intensity value of each point in the relative coordinate system. The relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system.

[0036] Finally, the point Normalize the intensity value of the point cloud, assuming that the maximum value of the pre-set point cloud intensity is The preset minimum value is , according to linear scaling so that the intensity value is between 0-1, that is: , so that each point is finally obtained The new coordinate values ​​and intensity values.

[0037] S30. Perform voxel division on the subsequence after the coordinate system conversion processing, evenly divide each frame of point cloud data into a number of preset voxel blocks, and encode the voxel blocks according to whether the voxel blocks contain valid points, so as to obtain valid voxels and empty voxels contained in each frame of point cloud data, wherein the voxel blocks containing valid points are valid voxels, and the voxel blocks containing valid points are empty voxels.

[0038] In this embodiment, step S30 specifically includes: S31, determine the valid area to be processed for each frame of point cloud data in each subsequence, and evenly divide the valid area into a number of preset voxel blocks, and determine the points under the point cloud data contained in each voxel block. The preset voxel block is a size set to Cube space, represents the size of the voxel block in the x-axis direction, represents the size of the voxel block in the y-axis direction, Indicates the size of the voxel block in the z-axis direction.

[0039] S32, performing voxel encoding on each voxel block using a preset encoder to determine valid voxels in the voxel block, wherein input data of the encoder is the relative coordinate system values ​​of all points in the voxel block.

[0040] Assume that the effective area RS to be processed by a subsequence is a A cubic space, where represents the size of the effective area RS in the x-axis direction, represents the size of the effective area RS in the y-axis direction, represents the size of the effective area RS in the z-axis direction. The sizes of a single preset voxel block in the x, y, and z-axis directions are , and , the space represented by the effective area RS can be evenly divided into several voxel blocks. After even division, the number of voxel blocks in the three directions of x, y, and z are respectively , and ,and = / , = / , = / ,total There are preset voxel blocks. Due to the sparse characteristics of point clouds, only some of these voxel blocks contain valid points, that is, valid voxels, and the remaining voxel blocks are empty, that is, empty voxels.

[0041] After completing the spatial voxel division, use the preset encoder to encode each valid voxel. The input data of the encoder is the coordinate value and intensity value of all points in a voxel block in the relative coordinate system, that is, the relative coordinate system value of the point. When the voxel block is a valid voxel, the encoder output data is a length of A vector of .

[0042] In this embodiment, the encoder may use static encoding in voxelnet or dynamic encoding in DynamicVoxel network structure. Figure 2 As shown, after completing the voxel encoding, the subsequence is transformed into voxels in space and the encoding of voxels. Figure 2 The gray part is the empty voxel, the green part is the valid voxel, and each valid voxel contains a length of Vector.

[0043] S40, extracting features from valid voxels in each frame of point cloud data and converting hollow voxels in each frame of point cloud data into dense voxel features to obtain a spatial feature vector for each frame of point cloud data, and obtaining a spatial feature vector of a subsequence based on the spatial feature vector for each frame of point cloud data.

[0044] In this embodiment, the typical structure of 3D sparse convolution in neural network is used to extract features from the previously generated voxel code. Each convolution only operates and outputs on valid voxels, which further improves the efficiency of point cloud data annotation. While continuously performing convolution, Stride convolution is used to gradually reduce the spatial size of the voxel feature map. The spatial size of the voxel feature map finally obtained is , and ,in, Represents the size of the voxel feature map in the x-axis direction, Represents the size of the voxel feature map in the y-axis direction, Indicates the size of the voxel feature map in the z-axis direction.

[0045] Then, by assigning 0 values ​​to all empty voxels, the sparse voxels are transformed into dense voxel features. Finally, the length of each voxel feature vector is , that is, each frame of point cloud data in this step subsequence obtains a size of The four-dimensional tensor of is denoted as the spatial eigenvector , use this tensor to represent the spatial voxel features of this frame of point cloud data.

[0046] S50, extracting the temporal feature vector of the subsequence using a neural network according to the spatial feature vector corresponding to each frame of point cloud data in the subsequence.

[0047] In this embodiment, step S50 specifically includes: S51 , evenly dividing the spatial feature vector corresponding to each frame of point cloud data into a plurality of sub-partitions, and determining the voxel blocks contained in each sub-partition.

[0048] S52, concatenating the feature vectors of all voxel blocks in the sub-partition forward and backward to obtain the subspace feature vector of the sub-partition.

[0049] S53, the subspace feature vectors of all subpartitions of the subsequence are collected, and time series feature extraction is performed on all subspace feature vectors contained in the subsequence through a neural network to obtain a time series feature vector.

[0050] In order to better extract the time series features and minimize the amount of calculation, this step first divides the spatial feature vector into subspaces. Specifically, it is evenly divided into sub-partitions, and the sizes of each sub-partition in the x, y, and z axes are , and .like Figure 3 As shown, Figure 3 Each color in represents a sub-partition. = / / / .

[0051] For each sub-partition, concatenate the feature vectors of all voxel blocks in the sub-partition to obtain a length of = Therefore, each frame of point cloud data can be obtained The length is The subspace eigenvector of . Figure 4 As shown, Figure 4 A subspace on the left is mapped to Figure 4 A voxel on the right, Figure 4 The length of the feature vector contained in each voxel on the left is , Figure 4 The length of the feature vector contained in each voxel on the right is .

[0052] A subsequence contains M frames of point cloud data, so a subsequence can generate subspace eigenvectors, the length of which is ,like Figure 5 As shown, is the time dimension, representing a subsequence containing M frames of point cloud data at different times. is the spatial dimension, which means that each frame has subspaces, the length of the eigenvector is , so the whole can be regarded as A three-dimensional tensor of .

[0053] like Figure 6 As shown, the three-dimensional tensor is transposed in the time dimension and space dimension and regarded as A time series with a size of M dimensions is constructed. Each M-dimensional time series is subjected to time series feature extraction through the Transformer or Recurrent Neural Network (RNN) in the neural network to obtain a feature vector (length ), and finally obtained The length is The time series feature vector of .

[0054] S60, perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform several convolution operations on the fused feature vector until a planar feature map with a size of 1 in the z-axis direction is formed, and generate an initial detection frame based on the planar feature map.

[0055] In this embodiment, the time series feature vector and spatial eigenvectors Perform feature splicing to obtain the fused feature vector , it can be understood that the fusion feature vector The size and spatial eigenvector of same.

[0056] Preferably, the feature concatenation method is: for the fused feature vector For any point in the space, first calculate the spatial feature vector according to the coordinates of the point Get the point space feature vector at the same coordinates of ), and then convert the coordinates according to Figure 4 The method of sub-partitioning and flattening mapping shown in the figure obtains the coordinate in the time series feature vector The point time series feature vector (the feature vector length is ), concatenate these two point vectors to get a new feature vector with a length of = + , and finally fused features The size is .

[0057] After obtaining the fusion feature vector of each frame of point cloud data After that, several 3D convolution operations are performed, in which the 3D convolution in the z-axis direction has a convolution step size of stide=2 until the size in the z-axis direction becomes 1, forming a plane feature map. According to this plane feature map, the initial detection frame is generated. The initial detection frame is generated by using several recurrent neural networks to sequentially output the center point heat map of the detection frame, classification category, target frame size, orientation and other parameters, so that each subsequence generates a corresponding detection frame.

[0058] The 3D convolution operation is actually a convolution operation performed in the x, y, and z directions, and the convolution step size stide=2 is set in the z-axis direction. In this way, each time a convolution is performed, the size in the z-axis direction will be reduced by half until the size in the z-axis direction becomes 1, which becomes a planar feature map.

[0059] S70 , continuously tracking the initial detection frames between the subsequences according to the spatial relationship between the initial detection frames between the subsequences to obtain a tracking trajectory.

[0060] The spatial relationship between the initial detection frames of the subsequences is obtained by using multi-target detection, and finally a set of tracking trajectories can be obtained. Each trajectory consists of a series of detection frames of different frames, representing the position of the same target object in different frames.

[0061] S80, adjusting the size of the initial detection frame according to the tracking trajectory to obtain a target detection frame.

[0062] In this embodiment, step S80 specifically includes: For each initial detection frame in the tracking trajectory, the tracking point cloud at the corresponding position is cropped from the corresponding frame in the subsequence, and then the coordinates of these tracking point clouds are converted to the detection frame coordinate system with the initial detection frame as the origin to obtain the converted point cloud. After that, the converted point clouds of all the initial detection frames in the tracking trajectory are superimposed to form a new adjusted point cloud. Next, the adjusted point cloud is input into the point cloud PointNet neural network to finally regress a finer size of the detection frame, that is, the target size. The generated target size is reassigned to all the initial detection frames in the trajectory to obtain a finer size result, that is, the target detection frame.

[0063] The detection boxes in the generated trajectory are converted into the format required by the annotation platform or algorithm and saved according to the requirements of the annotation platform. The subsequent annotation platform can directly read the data to display the annotation results, and algorithm researchers can directly read the format for training or testing.

[0064] The neural network-based point cloud automatic labeling method of the present invention can make full use of the spatial and temporal features of the point cloud. The accuracy and recall of the detection results generated based on the fusion of these two features are significantly improved. By optimizing the size of the original point cloud data based on the tracking results, a more accurate detection frame size can be obtained. Since the local features of the detected object have temporal continuity, when extracting the temporal features, the method of first dividing the subspace and then extracting is adopted, rather than calculating and extracting in the entire space, which can greatly reduce the complexity of the calculation.

[0065] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0066] See also Figure 7 , Figure 7 A schematic diagram of the structure of a point cloud automatic annotation device based on a neural network according to an embodiment of the present invention is shown. The device may include: The sequence division module 10 is used to obtain the original point cloud sequence and divide the point cloud sequence into a plurality of subsequences, wherein each subsequence contains a plurality of continuous frames of point cloud data.

[0067] First, get the original point cloud continuous sequence , They are the original point cloud data collected at N consecutive moments from 1 to N. Afterwards, the original point cloud continuous sequence is divided into several subsequences, each of which contains at most M consecutive frames of point cloud data, that is, M frames of point cloud data at consecutive moments.

[0068] In this embodiment, the subsequences are divided in such a way that M continuous frames of data are extracted from the original point cloud continuous sequence in order of time to obtain a subsequence. If enough M frames of point cloud data cannot be extracted in the last extraction, the actual number of remaining frames is used as the basis for extraction, thereby obtaining M subsequences.

[0069] For any subsequence ,in For the subsequence Frame, it can be seen The CCP has Frame point cloud data, For the subsequence Frame point cloud data.

[0070] The coordinate conversion module 20 is used to convert each point in each frame of point cloud data from the world coordinate system to the relative coordinate system according to the posture of each frame of point cloud data in the world coordinate system, and obtain the relative coordinate system value of each point, wherein the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system.

[0071] In this embodiment, each frame of point cloud data in the acquired original point cloud continuous sequence corresponds to the position and posture of the frame of point cloud data relative to the world coordinate system. , and, posture is a 4×4 coordinate transformation matrix. Then for the subsequences obtained by division In this step, you need to put The point cloud coordinate system is transferred to the origin set as The relative coordinate system is used and the intensity normalization operation is performed.

[0072] Specifically, assuming For the The first Frame point cloud data, its position in the world coordinate system is ,for Any point in the point cloud , it is necessary to set this point based on the pose Translate to Origin Set to (No. The relative coordinate value of the relative coordinate system of the first frame of point cloud data in the subsequence) ,in, Indicate point The coordinate in the x-axis direction in the relative coordinate system, Indicate point The coordinate in the y-axis direction in the relative coordinate system, Indicate point The coordinate in the z-axis direction in the relative coordinate system, Indicate point Intensity value in relative coordinate system (scale factor).

[0073] The voxel encoding module 30 is used to evenly divide each frame of point cloud data into a number of preset voxel blocks, and encode the voxel blocks according to whether the voxel blocks contain valid points, so as to obtain valid voxels and empty voxels contained in each frame of point cloud data, wherein the voxel blocks containing valid points are valid voxels, and the voxel blocks not containing valid points are empty voxels.

[0074] The spatial feature module 40 is used to extract features from valid voxels in each frame of point cloud data and convert empty voxels into dense voxel features to obtain a spatial feature vector for each frame of point cloud data, and obtain a subsequence spatial feature vector based on the spatial feature vector for each frame of point cloud data.

[0075] In this embodiment, the typical structure of 3D sparse convolution in neural network is used to extract features from the previously generated voxel code. Each convolution only operates and outputs on valid voxels, which further improves the efficiency of point cloud data annotation. While continuously performing convolution, Stride convolution is used to gradually reduce the spatial size of the voxel feature map. The spatial size of the voxel feature map finally obtained is , and ,in, Represents the size of the voxel feature map in the x-axis direction, Represents the size of the voxel feature map in the y-axis direction, Indicates the size of the voxel feature map in the z-axis direction.

[0076] Then, the sparse voxels are transformed into dense voxel features by assigning 0 values ​​to all empty voxels. Finally, the length of each voxel feature vector is , that is, each frame of point cloud data in this step subsequence obtains a size of The four-dimensional tensor of is denoted as the spatial eigenvector , use this tensor to represent the spatial voxel features of this frame of point cloud data.

[0077] The time series feature module 50 is used to extract the time series feature vector of the subsequence by using a neural network according to the spatial feature vector corresponding to each frame of point cloud data in the subsequence.

[0078] The feature fusion module 60 is used to perform feature fusion on the spatial feature vectors and the temporal feature vectors of the subsequence to obtain a fused feature vector, perform convolution processing on the fused feature vector in the z-axis direction until a planar feature map with a size of 1 in the z-axis direction is formed, and generate an initial detection frame based on the planar feature map.

[0079] In this embodiment, the time series feature vector and spatial eigenvectors Perform feature splicing to obtain the fused feature vector , it can be understood that the fusion feature vector The size and spatial eigenvector of same.

[0080] Preferably, the feature concatenation method is: for the fused feature vector For any point in the space, first calculate the spatial feature vector according to the coordinates of the point Get the point space feature vector at the same coordinates of ), and then convert the coordinates according to Figure 4 The method of sub-partitioning and flattening mapping shown in the figure obtains the coordinate in the time series feature vector The point time series feature vector (the feature vector length is ), concatenate these two point vectors to get a new feature vector with a length of = + , and finally fused features The size is .

[0081] After obtaining the fusion feature vector of each frame of point cloud data After that, several 3D convolution operations are performed, in which the 3D convolution in the z-axis direction has a convolution step size of stide=2 until the size in the z-axis direction becomes 1, forming a plane feature map. According to this plane feature map, the initial detection frame is generated. The initial detection frame is generated by using several recurrent neural networks to sequentially output the center point heat map of the detection frame, classification category, target frame size, orientation and other parameters, so that each subsequence generates a corresponding detection frame.

[0082] The 3D convolution operation is actually a convolution operation performed in the x, y, and z directions, and the convolution step size stide=2 is set in the z-axis direction. In this way, each time a convolution is performed, the size in the z-axis direction will be reduced by half until the size in the z-axis direction becomes 1, which becomes a planar feature map.

[0083] The trajectory tracking module 70 is used to continuously track the initial detection frames between the subsequences according to the spatial relationship between the initial detection frames between the subsequences to obtain a tracking trajectory.

[0084] The spatial relationship between the initial detection frames of the subsequences is obtained by using multi-target detection, and finally a set of tracking trajectories can be obtained. Each trajectory consists of a series of detection frames of different frames, representing the position of the same target object in different frames.

[0085] The size optimization module 80 is used to adjust the size of the initial detection frame according to the tracking trajectory to obtain the target detection frame.

[0086] The detection boxes in the generated trajectory are converted into the format required by the annotation platform or algorithm and saved according to the requirements of the annotation platform. The subsequent annotation platform can directly read the data to display the annotation results, and algorithm researchers can directly read the format for training or testing.

[0087] The neural network-based point cloud automatic annotation device of the present invention can make full use of the spatial features and temporal features of the point cloud. The accuracy and recall rates of the detection results generated based on the fusion of these two features are significantly improved. By optimizing the size of the original point cloud data based on the tracking results, a more accurate detection frame size can be obtained. Since the local features of the detected object have temporal continuity, when extracting the temporal features, the method of first dividing the subspace and then extracting is adopted, rather than calculating and extracting in the entire space, which can greatly reduce the complexity of the calculation.

[0088] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810 (processor), a communication interface 820 (Communications Interface), a memory 830 (memory) and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic command in the memory 830 to execute the point cloud automatic annotation method based on the neural network, and the method includes: Obtain the original point cloud sequence, and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; Each frame of point cloud data after conversion is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks without valid points are empty voxels; Extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence based on the spatial feature vector of each frame of point cloud data; Extract the temporal feature vector of the subsequence using neural network; Perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; According to the spatial relationship between the initial detection frames of the subsequences, the initial detection frames between the subsequences are continuously tracked to obtain the tracking trajectory; The size of the initial detection frame is adjusted according to the tracking trajectory to obtain the target detection frame.

[0089] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0090] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the neural network-based point cloud automatic annotation method provided by the above methods, the method comprising: Obtain the original point cloud sequence, and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; Each frame of point cloud data after conversion is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks without valid points are empty voxels; Extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence based on the spatial feature vector of each frame of point cloud data; Extract the temporal feature vector of the subsequence using neural network; Perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; According to the spatial relationship between the initial detection frames of the subsequences, the initial detection frames between the subsequences are continuously tracked to obtain the tracking trajectory; The size of the initial detection frame is adjusted according to the tracking trajectory to obtain the target detection frame.

[0091] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned method for automatically annotating a point cloud based on a neural network, the method comprising: Obtain the original point cloud sequence, and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; Each frame of point cloud data after conversion is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks without valid points are empty voxels; Extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence based on the spatial feature vector of each frame of point cloud data; Extract the temporal feature vector of the subsequence using neural network; Perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; According to the spatial relationship between the initial detection frames of the subsequences, the initial detection frames between the subsequences are continuously tracked to obtain the tracking trajectory; The size of the initial detection frame is adjusted according to the tracking trajectory to obtain the target detection frame.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud automatic annotation method based on neural network, characterized in that: The method comprises: Obtain the original point cloud sequence, and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; Each frame of point cloud data after conversion is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks without valid points are empty voxels; Extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence based on the spatial feature vector of each frame of point cloud data; Extract the temporal feature vector of the subsequence using neural network; Perform feature fusion on the spatial feature vector and the temporal feature vector of the subsequence to obtain a fused feature vector, perform convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; According to the spatial relationship between the initial detection frames of the subsequences, the initial detection frames between the subsequences are continuously tracked to obtain the tracking trajectory; The size of the initial detection frame is adjusted according to the tracking trajectory to obtain the target detection frame.

2. The method for automatic point cloud annotation based on neural network according to claim 1, characterized in that: According to the pose of each frame of point cloud data in the subsequence in the world coordinate system, each point in each frame of point cloud data is converted from the world coordinate system to the relative coordinate system to obtain the relative coordinate system value of each point, specifically including: Determine the pose of each frame of point cloud data in the subsequence relative to the world coordinate system, and convert each point under each point cloud data in the subsequence into homogeneous coordinates; According to the homogeneous coordinates corresponding to each point, each point in the subsequence is converted from the world coordinate system to the relative coordinate system; According to the point cloud data corresponding to each point and the position of the first frame of the subsequence point cloud data relative to the world coordinate system, the coordinate values ​​in the aligned secondary coordinates are converted to determine the coordinate value of each point in the relative coordinate system; According to the preset maximum and minimum values ​​of the point cloud intensity, the intensity values ​​in the secondary coordinates are normalized to determine the intensity value of each point in the relative coordinate system.

3. The method for automatic point cloud annotation based on neural network according to claim 1, characterized in that: The converted point cloud data of each frame is evenly divided into a number of preset voxel blocks, and the voxel blocks are encoded according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data, specifically including: Determine the effective area to be processed for each frame of point cloud data in each subsequence, and evenly divide the effective area into a number of preset voxel blocks, and determine the points under the point cloud data contained in each voxel block; the preset voxel block is set to a size of Cube space, represents the size of the voxel block in the x-axis direction, represents the size of the voxel block in the y-axis direction, Indicates the size of the voxel block in the z-axis direction; Each voxel block is voxel-encoded using a preset encoder to determine valid voxels in the voxel block; the input data of the encoder is the relative coordinate system values ​​of all points in the voxel block.

4. The method for automatic point cloud annotation based on neural network according to claim 3, characterized in that: The extracting of the time series feature vector of the subsequence by using a neural network specifically includes: The spatial feature vector corresponding to each frame of point cloud data is evenly divided into several sub-partitions, and the voxel blocks contained in each sub-partition are determined; Concatenate the feature vectors of all voxel blocks in the subpartition forward and backward to obtain the subspace feature vector of the subpartition; The subspace feature vectors of all subpartitions of the subsequence are collected, and the time series features of all subspace feature vectors contained in the subsequence are extracted through a neural network to obtain a time series feature vector.

5. The method for automatic point cloud annotation based on neural network according to claim 1, characterized in that: The feature fusion method is as follows: Obtain the point spatial feature vector of each point in the spatial feature vector and the point temporal feature vector of each point in the temporal feature vector, and concatenate the point spatial feature vector and the point temporal feature vector to obtain a point fusion vector; The point fusion vectors of each point in the subsequence are collected to obtain the feature fusion vector.

6. The method for automatic point cloud annotation based on neural network according to claim 1, characterized in that: Converting empty voxels to dense voxel features is achieved by assigning 0 values ​​to all empty voxels.

7. The method for automatic point cloud annotation based on neural network according to claim 1, characterized in that: The step of adjusting the size of the initial detection frame according to the tracking trajectory to obtain the target detection frame specifically includes: Using each initial detection frame in the tracking trajectory, a tracking point cloud at a corresponding position is cut out from the corresponding frame in the subsequence; Convert the coordinates of the tracking point cloud to the detection frame coordinate system with the initial detection frame as the origin to obtain the converted point cloud; Superimpose the transformed point clouds of all initial detection boxes in the trajectory to form an adjusted point cloud; Input the adjusted point cloud into the point cloud PointNet neural network for regression processing, and the target size is obtained by the output of the point cloud PointNet neural network; Reassign the target size to all initial detection boxes in the tracking trajectory to obtain the target detection box.

8. A point cloud automatic annotation device based on a neural network, characterized in that: The device comprises: The sequence division module is used to obtain the original point cloud sequence and divide the point cloud sequence into several subsequences; each subsequence contains several consecutive frames of point cloud data; The coordinate conversion module is used to convert each point in each frame of point cloud data from the world coordinate system to the relative coordinate system according to the position of each frame of point cloud data in the subsequence in the world coordinate system, and obtain the relative coordinate system value of each point; the origin of the relative coordinate system is the first frame of point cloud data in the corresponding subsequence, and the relative coordinate system value includes the coordinate value and intensity value of each point in the relative coordinate system; The voxel encoding module is used to evenly divide each frame of point cloud data after conversion into a number of preset voxel blocks, and encode the voxel blocks according to the valid points in the voxel blocks to obtain the valid voxels and empty voxels contained in each frame of point cloud data; the voxel blocks containing valid points are valid voxels, and the voxel blocks not containing valid points are empty voxels; The spatial feature module is used to extract features from valid voxels and convert empty voxels into dense voxel features to obtain the spatial feature vector of each frame of point cloud data, and obtain the spatial feature vector of the subsequence according to the spatial feature vector of each frame of point cloud data; A time series feature module is used to extract the time series feature vector of the subsequence using a neural network; A feature fusion module is used to fuse the spatial feature vectors and the temporal feature vectors of the subsequences to obtain a fused feature vector, perform a convolution operation on the fused feature vector to obtain a plane feature map, and generate an initial detection frame based on the plane feature map; A trajectory tracking module is used to continuously track the initial detection frames between the subsequences according to the spatial relationship between the initial detection frames between the subsequences to obtain a tracking trajectory; The size optimization module is used to adjust the size of the initial detection frame according to the tracking trajectory to obtain the target detection frame.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the point cloud automatic labeling method based on a neural network as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatic point cloud annotation based on a neural network as described in any one of claims 1 to 7 are implemented.

Images