Obstacle detection method and device

Abstract

According to the obstacle detection method and device, during obstacle detection, collected data are point cloud data collected by a millimeter-wave radar, the millimeter-wave radar is high in fog, smoke and dust penetrating capacity, small in influence of environment changes and extreme weather and high in anti-interference capacity, and therefore the obstacle can be detected in the severe weather. When the point cloud data collected by the millimeter wave radar is used for obstacle detection, the detection accuracy, stability and robustness can be improved. Besides, after the point cloud feature information is converted into the pseudo-image features, the pseudo-image features are further subjected to feature extraction operation, so that the obtained multi-scale features are richer and can better represent the features of the obstacles, and the accuracy of subsequent obstacle recognition can be improved; and the requirements in automatic driving practical application tasks are better met.

Description

Technical Field

[0001] The present invention relates to the fields of computers and autonomous driving, and particularly to the fields of deep learning, 3D point cloud processing, and object detection. More specifically, it relates to an obstacle detection method and device. Background Art

[0002] 3D object detection is extremely important in the field of autonomous driving. In autonomous driving scenarios, many tasks such as mission decision-making and path planning require the vehicle to have good perception and expression capabilities of the surrounding 3D space. Therefore, accurately detecting obstacle information around the vehicle from the data obtained by sensors has important practical significance for autonomous driving safety, autonomous parking, perception positioning, and road obstacle avoidance.

[0003] Currently, when performing 3D obstacle detection, data collected by lidar is generally used for obstacle detection operations. Lidar has weak anti-interference ability, is easily affected by environmental changes and extreme weather, and its ability to penetrate fog, smoke, dust, etc. is also relatively weak. Therefore, in bad weather, the accuracy of the data collected by lidar is poor, resulting in poor accuracy of 3D obstacle detection. Summary of the Invention

[0004] In view of this, the present invention provides an obstacle detection method and device to solve the problem of poor accuracy of 3D obstacle detection using lidar.

[0005] To solve the above technical problems, the present invention adopts the following technical solutions:

[0006] An obstacle detection method, comprising:

[0007] Obtaining point cloud data collected by a millimeter-wave radar, and determining point cloud feature information of the point cloud data;

[0008] Converting the point cloud feature information into pseudo-image features;

[0009] Performing a feature extraction operation on the pseudo-image features to obtain multi-scale features;

[0010] Performing an obstacle recognition operation based on the multi-scale features to obtain an obstacle detection result of the point cloud data.

[0011] Optionally, determining the point cloud feature information of the point cloud data includes:

[0012] Calculating the position coordinates of each point in the point cloud data;

[0013] Determining the attribute features of each point, where the attribute features include relative velocity, radar cross section, Doppler information, and echo intensity;

[0014] Take the position coordinates and the attribute features as the point cloud feature information.

[0015] Optionally, convert the point cloud feature information into pseudo-image features, including:

[0016] Perform a columnar operation on the point cloud feature information to compress the height information to obtain pseudo-image features; the pseudo-image features include a plurality of pillars.

[0017] Optionally, perform a feature extraction operation on the pseudo-image features to obtain multi-scale features, including:

[0018] Call the U-Net model to enable the U-Net model to perform a feature extraction operation on the pseudo-image features to obtain multi-scale features.

[0019] Optionally, perform an obstacle recognition operation based on the multi-scale features to obtain the obstacle detection result of the point cloud data, including:

[0020] Input the multi-scale features into a semantic segmentation head for an obstacle category detection operation to obtain an obstacle category detection result;

[0021] Input the multi-scale features into an orientation information regression head for an orientation information detection operation to obtain an obstacle orientation information detection result;

[0022] Input the multi-scale features into an instance segmentation head for an instance detection operation to obtain an instance classification result;

[0023] Fuse the obstacle category detection result, the obstacle orientation information detection result, and the instance classification result to obtain an obstacle detection result.

[0024] Optionally, the semantic segmentation head, the orientation information regression head, and the instance segmentation head are trained based on sample data; the training process of the semantic segmentation head, the orientation information regression head, and the instance segmentation head includes:

[0025] Obtain sample data, where the sample data includes a point cloud data sample configured with point cloud feature information and label information; the label information is the obstacle category label used for training the semantic segmentation head, the orientation label used for training the orientation information regression head, and the instance label used for training the instance segmentation head;

[0026] Determine the multi-scale feature data corresponding to the point cloud feature information;

[0027] Use the multi-scale feature data and the label information to train the semantic segmentation head, the orientation information regression head, and the instance segmentation head respectively until the training stop condition is met and then stop.

[0028] Optionally, after performing an obstacle recognition operation based on the multi-scale features to obtain the obstacle detection result of the point cloud data, the following steps are further included:

[0029] Determine the detection boxes of the pillars in the obstacle detection result;

[0030] Count the number of pillars located in the detection boxes, and based on the number, determine the scores of the detection boxes;

[0031] Based on the scores of the detection boxes, determine the bounding boxes of each obstacle in the obstacle detection result from the detection boxes.

[0032] Optionally, determining the detection boxes of the pillars in the obstacle detection result includes:

[0033] Obtain a detection box generation rule, where the detection box generation rule includes the relative position relationship between the pillar and the detection box;

[0034] According to the detection box generation rule, generate the detection boxes of each pillar in the obstacle detection result.

[0035] Optionally, based on the scores of the detection boxes, determining the bounding boxes of each obstacle in the obstacle detection result from the detection boxes includes:

[0036] Use the scores to perform a non-maximum suppression operation on the detection boxes to obtain the bounding boxes of each obstacle in the obstacle detection result.

[0037] An obstacle detection device includes:

[0038] A feature determination module, configured to obtain the point cloud data collected by a millimeter-wave radar and determine the point cloud feature information of the point cloud data;

[0039] A feature processing module, configured to convert the point cloud feature information into pseudo-image features;

[0040] A feature extraction module, configured to perform a feature extraction operation on the pseudo-image features to obtain multi-scale features;

[0041] An obstacle detection module, configured to perform an obstacle recognition operation based on the multi-scale features to obtain the obstacle detection result of the point cloud data.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention provides an obstacle detection method and device. In the present invention, when performing obstacle detection, the collected data is point cloud data collected by a millimeter-wave radar. The millimeter-wave radar has strong ability to penetrate fog, smoke, and dust, is less affected by environmental changes and extreme weather, and has strong anti-interference ability. Therefore, when using the point cloud data collected by the millimeter-wave radar for obstacle detection in bad weather, the detection accuracy can be improved. In addition, in the present invention, after converting the point cloud feature information into pseudo-image features, further feature extraction operations are performed on the pseudo-image features, so that the obtained multi-scale features are richer and more representative of the features of obstacles, and the accuracy of subsequent obstacle recognition will also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0045] Figure 1 It is a flowchart of a method for an obstacle detection method provided by an embodiment of the present invention;

[0046] Figure 2 It is a diagram showing the representation of strut data provided by an embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of a pseudo-image provided by an embodiment of the present invention;

[0048] Figure 4 It is a structural diagram of a U-Net model provided by an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of a detection head provided by an embodiment of the present invention;

[0050] Figure 6 It is a flowchart of another obstacle detection method provided by an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of a detection frame provided by an embodiment of the present invention;

[0052] Figure 8 It is a schematic diagram of sample annotation provided by an embodiment of the present invention;

[0053] Figure 9 It is a logic diagram of network training provided by an embodiment of the present invention;

[0054] Figure 10A schematic diagram of the scenario of an obstacle detection method provided by an embodiment of the present invention;

[0055] Figure 11 A schematic structural diagram of an obstacle detection device provided by an embodiment of the present invention. Detailed implementation manners

[0056] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0057] 3D object detection is extremely important in the field of autonomous driving. In the autonomous driving scenario, many tasks such as task decision-making and path planning require the vehicle to have good perception and expression capabilities of the surrounding 3D space. Therefore, accurately detecting the obstacle information around the vehicle through the data obtained by the sensor has important practical significance for the safety of autonomous driving, automatic parking, perception positioning, and road obstacle avoidance.

[0058] Currently, when performing 3D obstacle detection, generally the data collected by lidar is used for obstacle detection operations. Lidar has weak anti-interference ability, is easily affected by environmental changes and extreme weather, and also has relatively weak ability to penetrate fog, smoke, dust, etc. Therefore, in bad weather, the accuracy of the data collected by lidar is poor, resulting in poor accuracy of 3D obstacle detection. Moreover, lidar equipment is expensive and the cost is high.

[0059] When using the data collected by lidar for obstacle detection, data fusion may also be involved, such as fusing lidar, ultrasonic radar, and image data for obstacle detection. In this way, although the accuracy of obstacle detection can be improved, such methods require calibrating the data corresponding to each sensor, matching frame by frame in time and space, and also require multiple detection models to process the data, resulting in too high cost and being difficult to apply to real scenarios.

[0060] In addition, using historical time series information to superimpose consecutive frames can indeed improve the performance of point cloud data and further improve the detection accuracy, but at the same time it will also make the data become more massive, resulting in an increase in the processing time required, which does not meet the requirement of quickly responding to obstacles in real-time road scenarios.

[0061] Therefore, it is urgent to improve the accuracy of obstacle detection, and while improving the accuracy, it is possible to improve the detection speed and reduce the cost.

[0062] After research, it is found that millimeter-wave radars, such as 4D imaging radars, have strong capabilities of penetrating fog, smoke, and dust, are less affected by environmental changes and extreme weather, and have strong anti-interference capabilities. Therefore, when using the point cloud data collected by millimeter-wave radars for obstacle detection, the detection accuracy can be improved.

[0063] Moreover, millimeter-wave radars are inexpensive, have low costs, and high feasibility.

[0064] In addition, when using a millimeter-wave radar for obstacle recognition, only performing obstacle detection operations based on the currently collected data of the millimeter-wave radar without the need for data superposition and fusion can reduce the amount of data processing, thereby improving the obstacle detection speed and meeting the requirement of faster response to obstacles in real-time road scenarios.

[0065] Specifically, the present invention performs 3D obstacle detection based on a forward-looking 4D imaging radar. Specifically, based on the point cloud collected by a calibrated millimeter-wave radar, such as a 4D imaging radar, installed on an automobile body, obstacles in front of the automobile are detected to obtain an obstacle detection result based on the spatial position of the point cloud, where the detection result includes the spatial position, its own size, and traveling direction of the obstacle.

[0066] On the basis of the above content, an embodiment of the present invention provides an obstacle detection method PillarSeg. Referring to Figure 1 , the obstacle detection method may include:

[0067] S11. Obtain the point cloud data collected by the millimeter-wave radar and determine the point cloud feature information of the point cloud data.

[0068] In practical applications, the millimeter-wave radar in this embodiment may be, for example, a 4D imaging radar. The 4D imaging radar may be a forward-looking 4D imaging radar, which is calibrated and installed on the front side of the automobile body to collect the point cloud data on the front side of the vehicle.

[0069] During the driving of the automobile, the forward-looking 4D imaging radar may be used to collect the point cloud data on the front side of the vehicle in real time, such as Radar point cloud data.

[0070] After the point cloud data is collected, it is necessary to determine the point cloud feature information of the point cloud data.

[0071] The point cloud feature information may include data such as the position coordinates, relative velocity, radar cross section, Doppler information, and echo intensity of each point. Among them, the relative velocity, radar cross section, Doppler information, and echo intensity are the attribute features of each point and need to be determined.

[0072] First, calculate the position coordinates of each point in the point cloud data. The distance information of the target needs to be obtained by performing a distance FFT (Fast Fourier Transform) operation on the initially sampled point cloud data. The distance calculation formula is:

[0073]

[0074] Δf is the frequency difference between the received signals of two adjacent chirp signals, λ is the speed of light, and S is the frequency change rate.

[0075] Based on the calculated r, convert it into the x, y, and z coordinates of each point in the Euclidean space. The x, y, and z coordinates are the position coordinates.

[0076] The relative velocity refers to the relative velocities v x 、v y of this point along the x and y axes in the Euclidean space. It is necessary to perform a velocity FFT (Fourier Transform) operation on the initially sampled point cloud data. The velocity calculation formula is:

[0077]

[0078] Among them, is the phase difference between the received signals of two adjacent chirp signals, λ is the speed of light, and Tc is the duration length.

[0079] After calculating v, v x 、v y .

[0080] The radar cross section (RCS), Doppler information doppler, and echo intensity power can be directly obtained from the point cloud data.

[0081] S12. Convert the point cloud feature information into pseudo-image features.

[0082] Specifically, through data preprocessing operations, perform a columnar operation on the point cloud feature information to compress the height information, thereby converting it into pseudo-image features. The pseudo-image features include multiple pillars Pillar and are used as the input of the subsequent network.

[0083] Among them, the columnar operation is equivalent to dividing a fixed-size grid on the X-Y plane of the point cloud space, extracting the features of the point cloud in each non-empty columnar grid, and obtaining a feature vector for each Pillar, thereby realizing the compression of the height information and obtaining the pseudo-image features.

[0084] In this embodiment, converting the point cloud feature information into pseudo-image features is because pillarization adapts to the sparsity of 4D imaging radar point cloud data, which can further improve the calculation efficiency. When deploying in an autonomous driving scenario, for the perception resolution of the environmental world, it is not necessary to reach the point or voxel level. Having a good perception ability at the Pillar level is sufficient, and the implementation is relatively simple. Pillarization, compared with the voxel-based method, no longer divides by height, but directly aggregates into pillars perpendicular to the ground.

[0085] Refer to Figure 2 , Figure 2 which is the pillar data representation. The specific conversion process is as follows:

[0086] First, divide the point cloud data into grids according to the x and y axes where the point cloud data is located. At the same time, consider all the point clouds falling within this grid as being in the same pillar (Pillar), or it can also be regarded as these points constituting this Pillar.

[0087] Each point cloud is represented by a D = 15-dimensional vector, which are respectively (x, y, z, r, d, p, v x , v y , v obj , x p , y p , z p , x f , y f , z f ), where x, y, z are the true three-dimensional coordinate information of each point cloud; r, d, p are the radar cross section (Radar Cross Section, RCS), doppler, and power of the point cloud; v x , v y is the relative velocity of the point cloud in the x and y directions; v obj is the absolute motion velocity of the point cloud. x p , y p , z p are respectively the offsets of each point cloud relative to the centroid coordinates of this Pillar in this coordinate system. The Pillar centroid is defined as where N is the number of non-zero points in this Pillar, (x p , y p , z p ) = (x - x pc , y - y pc , z - z pc ), x f , y f , z fThe offset of each point cloud relative to the center coordinates of the Pillar in this coordinate system.

[0088] To ensure data consistency and facilitate subsequent calculations, a hyperparameter N needs to be set to limit the number of point clouds within each Pillar. If the number of point cloud data in a Pillar exceeds N, it is randomly sampled to N; if the number of point cloud data in a Pillar is less than N, the missing part is filled by padding with zeros. In this method, N = 16 is set.

[0089] Thus, assuming there are P non-empty Pillars in the sample, this sample can be represented by a tensor of dimension (D, P, N).

[0090] After tensorization is achieved, feature extraction can be performed on the point cloud data. The main process is as follows: First, the feature dimension of the point cloud is processed through a certain feature extraction layer, which is transformed from (D, P, N) to (C, P, N). In this method, C = 64, and a max pooling operation is performed on it in the Pillar dimension. Thus, a tensor of dimension (C, P) is obtained as the feature map. Then, the feature map needs to be transformed into the form of a pseudo-image to provide spatial dependency information for the subsequent network model. The specific method is to transform P into (H, W) according to the spatial position of the Pillars, that is, P → (H, W). Thus, a pseudo-image of dimension (C, H, W) can be obtained, and its conversion process is as Figure 3 shown. Among them, H and W are the sizes of the X-Y plane after pillarization.

[0091] S13. Perform feature extraction operations on the pseudo-image features to obtain multi-scale features.

[0092] In this embodiment, to further enrich the features, further feature extraction operations are performed on the pseudo-image features.

[0093] When performing feature extraction, since the U-Net model is an encoder-decoder structure, which is simple and computationally efficient, in this embodiment, the U-Net model is used as the feature extraction network. Then, in the actual use process, the U-Net model is called to enable the U-Net model to perform feature extraction operations on the pseudo-image features to obtain multi-scale features.

[0094] Specifically, the U-Net model is a segmentation model, and the overall model structure is U-shaped. The specific structure refers to Figure 6 , and the pseudo-image features obtained in the above step S12 are input into the U-Net model, and multi-scale features can be output.

[0095] It should be noted that, if in the actual obstacle detection task, the detection accuracy requirement is not high but the detection speed requirement is high, step S13 may not be executed, and step S14 may be directly executed based on the pseudo image features to reduce the feature extraction time and improve the detection efficiency.

[0096] S14. Perform an obstacle recognition operation based on the multi-scale features to obtain an obstacle detection result of the point cloud data.

[0097] After obtaining richer multi-scale features, obstacle recognition operations can be performed. Since the model is trained based on a large number of samples and has high detection accuracy, the model is used in this embodiment to implement obstacle recognition operations.

[0098] The model can be a multi-task detection head. In order to be applicable to the obstacle detection scenario in this embodiment, three network output heads are used to detect the category, orientation and instance of the obstacle.

[0099] Specific reference Figure 5 , the three network output heads are semantic segmentation head, orientation information regression head and instance segmentation head, which can all be implemented through full convolutional layers, such as 2D convolutional layers.

[0100] The size of the semantic segmentation head is 8×W (width)×H (height), which is used for obstacle category detection, specifically to detect the obstacle category to which the Pillar in each multi-scale feature belongs. This algorithm defines a total of 8 obstacle categories, namely background, car, truck, bus, pedestrian, motorcycle, bicycle and tricycle. Each Pillar can get its category through this segmentation head.

[0101] The size of the orientation information regression head is 1×W (width)×H (height), which directly regresses the orientation information of each Pillar. Through this regression head, the specific orientation angle prediction value of each Pillar can be obtained.

[0102] The instance segmentation head size is 50×W (width)×H (height), which is used to distinguish obstacle Pillars that belong to the same category but not the same instance, such as distinguishing each of three vehicles. This task can be converted into a classification problem. In this algorithm, by counting the distribution of real instances, a total of 50 categories are defined according to the distance from near to far (the first closest target, the second closest target, the third closest target...), and after a set of classification convolutional layers, the category to which each obstacle Pillar belongs can be obtained, thereby achieving the purpose of instance segmentation.

[0103] Based on the above semantic segmentation head, orientation information regression head, and instance segmentation head, the multi-scale features are input into the semantic segmentation head for obstacle category detection operations to obtain obstacle category detection results. The multi-scale features are input into the orientation information regression head for orientation information detection operations to obtain obstacle orientation information detection results. The multi-scale features are input into the instance segmentation head for instance detection operations to obtain instance classification results.

[0104] After obtaining the corresponding detection results through the three detection heads, since the detection results are independent of each other, in this embodiment, a fusion operation needs to be performed on them, that is, fusing the obstacle category detection results, the obstacle orientation information detection results, and the instance classification results to obtain the final obstacle detection results.

[0105] When performing the fusion, first, each obstacle detected in the obstacle category detection results can be determined. Using the obstacle orientation information detection results, its orientation can be determined. Then, using the instance classification results, its instance can be determined. Finally, the category, orientation, and instance classification of each obstacle can be obtained, that is, the obstacle detection results.

[0106] In practical applications, if the obstacle detection results show obstacles in the form of Pillars, then the category, orientation, and instance classification of each Pillar can be determined.

[0107] In this embodiment, when performing obstacle detection, the data collected is the point cloud data collected by a millimeter-wave radar. The millimeter-wave radar has strong penetration ability for fog, smoke, and dust, is less affected by environmental changes and extreme weather, and has strong anti-interference ability. Therefore, when using the point cloud data collected by the millimeter-wave radar for obstacle detection, the detection accuracy can be improved, and it is beneficial to maintain the stability of the system in different weather environments. In addition, in the present invention, after converting the point cloud feature information into pseudo-image features, further feature extraction operations are performed on the pseudo-image features to make the obtained multi-scale features richer and more representative of the features of obstacles, which will also improve the accuracy of subsequent obstacle recognition.

[0108] In addition, the present invention is different from traditional point cloud-based deep learning 3D object detection algorithms. This method no longer predicts rectangular bounding boxes based on contour points, but directly determines whether each point cloud cluster is an obstacle and what kind of obstacle it is, and can perform post-processing on the detected obstacles to form a bounding box, so as to achieve the purpose of obstacle detection, and it is also more in line with the requirements in the actual application tasks of autonomous driving. This algorithm can be applied to the front-view obstacle detection task of autonomous vehicles.

[0109] In addition, the embodiments of the present invention abandon the inefficient and complex preset anchor boxes, and utilize the characteristics of the pseudo-image, and use 2D convolution operations instead of 3D convolution operations used by most methods, which greatly improves the calculation speed of the algorithm.

[0110] The above embodiments introduce how to perform obstacle detection. In another implementation manner of the present invention, since the obstacles in the above obstacle detection results are presented in the form of Pillars, and one obstacle may include multiple Pillars, in order to be able to visually display the final determined obstacles in the obstacle detection results, the bounding boxes of the obstacles can be marked in the obstacle detection results to visually display the detected obstacles.

[0111] Then after step S14, referring to Figure 6 , it may further include:

[0112] S25. Determine the detection boxes of the pillars in the obstacle detection results.

[0113] In practical applications, the detection box can be a 3D bounding box for displaying the boundary of the obstacle. In this embodiment, a corresponding 3D bounding box needs to be generated according to the final obstacle detection result in this algorithm, that is, the segmented result is framed by the detection box.

[0114] In the obstacle detection results, the obstacles exist in the form of at least one Pillar, so it is necessary to infer the boundary of the obstacle based on each Pillar. In practical applications, each Pillar is a part of the obstacle. In order to detect the boundary of the obstacle as much as possible, it can be assumed that it is a boundary point of the obstacle bounding box. Taking the obstacle as a vehicle as an example, the bounding box is placed based on this boundary point, and the bounding box represents the possible vehicle position situation.

[0115] Since in practice, more attention is paid to the position of the obstacle closer to the host vehicle, therefore, in this algorithm, each Pillar corresponds to the following eight situations, and the visualization effect is as Figure 7 shown:

[0116] 1) This Pillar is the midpoint of the rear of the vehicle moving forward straight;

[0117] 2) This Pillar is the midpoint of the rear of the vehicle moving forward and skewed to the left;

[0118] 3) This Pillar is the midpoint of the rear of the vehicle moving forward and skewed to the right;

[0119] 4) This Pillar is the midpoint of the side of the vehicle moving forward horizontally and close to the host vehicle;

[0120] 5) This Pillar is the corner point close to the host vehicle on the right side of the vehicle moving forward straight;

[0121] 6) The Pillar is the corner point approaching the host vehicle on the left side of the vehicle directly ahead.

[0122] 7) The Pillar is the corner point approaching the host vehicle of the vehicle skewed leftward in the forward direction.

[0123] 8) The Pillar is the corner point approaching the host vehicle of the vehicle skewed rightward in the forward direction.

[0124] The above eight situations are the detection box generation rules. The detection box generation rules include the relative position relationship between the pillar and the detection box. For example, in the first situation, the pillar is at the midpoint of the tail of the detection box. In the second situation, the pillar is at the midpoint of the tail of the vehicle skewed rightward in the direction.

[0125] According to the above detection box generation rules, the detection boxes of each pillar in the obstacle detection result can be generated. Specifically, it can be referred to Figure 7 , for example, by placing the pillar at the midpoint of the tail of the detection box, a detection box in the left middle can be formed. The formation methods of the other detection boxes are similar. Figure 7 The boundary boxes of the obstacles that each pillar may correspond to are shown as much as possible in

[0126] Figure 7 . Subsequently, the final boundary box of each obstacle needs to be selected from the above multiple detection boxes.

[0127] S26. Count the number of pillars located in the detection box, and based on this number, determine the score of the detection box.

[0128] When selecting the final boundary box of each obstacle from the above multiple detection boxes, considering that if a detection box is an obstacle, such as the boundary box of a vehicle, the number of Pillars included in it will be relatively large. Then, the boundary box can be screened based on the number of pillars in the detection box.

[0129] In this embodiment, it is necessary to count the number of pillars located in the detection box. When counting the number, all detection boxes need to be traversed. As long as at least a part of a certain Pillar is included, it will be counted. Finally, the number of pillars in each detection box is obtained. The corresponding score is obtained by calculating the number of Pillars delimited in the detection box. The more Pillars contained in the box, the higher the score. The specific score calculation rule can be configured according to the actual scenario, such as the score is the number of Pillars.

[0130] S27. Determine the boundary boxes of each obstacle in the obstacle detection result from the detection box.

[0131] In this embodiment, since eight detection frames are defined for each Pillar, there are multiple detection frames corresponding to one obstacle, that is, there is redundancy in the detection frames. In this step, the redundant detection frames are deleted so that only one detection frame is retained for one obstacle.

[0132] In the specific technical implementation, the score is used to perform non-maximum suppression (NMS) operation on the detection frames to delete the redundant detection frames, thereby obtaining the bounding boxes of each obstacle in the obstacle detection result.

[0133] In this embodiment, after the obstacle is detected, the bounding box of the obstacle is further determined, so that the position and category of each obstacle can be intuitively viewed, which is convenient for the user to view, and provides an obstacle basis for subsequent path planning and other processing operations.

[0134] It should be noted that for steps S21 - S24 in this embodiment, please refer to the corresponding descriptions in the above embodiments.

[0135] The above embodiment uses a detection head for obstacle detection. In practical applications, the above semantic segmentation head, the orientation information regression head, and the instance segmentation head are trained based on sample data. This embodiment introduces the training processes of the semantic segmentation head, the orientation information regression head, and the instance segmentation head. Specifically, the training process includes the following steps:

[0136] 1) Obtain sample data.

[0137] In practical applications, the sample data includes point cloud data samples configured with point cloud feature information and label information; the label information is the obstacle category label used for training the semantic segmentation head, the orientation label used for training the orientation information regression head, and the instance label used for training the instance segmentation head.

[0138] First, before model training, data collection and annotation operations need to be carried out. During collection, a millimeter-wave radar, such as a 4D imaging radar, is used to operate on the point cloud data. For the convenience of subsequent annotation, the vehicle's camera can be used to collect images convenient for the vehicle, and the point cloud data can be annotated with the assistance of the image data.

[0139] In this embodiment, during annotation, manual annotation can be adopted. The annotation categories include a total of eight obstacle categories: background, sedan, truck, bus, pedestrian, motorcycle, bicycle, and tricycle. These categories are the obstacle category labels used for training the semantic segmentation head. These categories can be expanded according to the actual situation to achieve the purpose of detecting a variety of obstacles. In addition, the orientation label needs to be determined, which can be based on the orientation of the annotated obstacles. This orientation label is the orientation label used for training the orientation information regression head. In addition, for the instance segmentation head, standard instance labels are required. The instance labels divide the obstacles into individual instances.

[0140] In addition to annotating its own category, orientation, and instance information for each point, nine features also need to be determined: the x, y, and z coordinates of the point in the Euclidean space, and the relative velocities v x 、v y along the x and y axes in the Euclidean space, the radar cross section (Radar Cross Section, RCS), doppler, and power. These nine features are the above-mentioned point cloud feature information.

[0141] That is, in this embodiment, the sample data includes point cloud data samples configured with point cloud feature information and label information. The label information is the obstacle category label used for training the semantic segmentation head, the orientation label used for training the orientation information regression head, and the instance label used for training the instance segmentation head.

[0142] During network training, the semantic segmentation head only uses the obstacle category label and does not use the orientation label and the instance label. The same applies to other categories.

[0143] Since manual annotation has the problem of low efficiency, to improve efficiency, experiments can be quickly conducted on the public dataset, and the points within the provided true bounding boxes can be directly labeled as obstacle points.

[0144] It should be noted that in the present invention, during obstacle detection, as long as the obstacles are detected, obtaining the final bounding box is only for display convenience. In actual applications, during network training, the samples used do not need to carry the bounding box. Therefore, when directly using the public dataset, the bounding box in it can be deleted and only the labeled points are used, or the bounding box can be retained, but it is not used during training.

[0145] Refer to Figure 8 This algorithm does not require Figure 8 the bounding box-level annotation shown on the left two, and directly performs Figure 8For the two point-level annotations shown on the right, currently, in order to quickly conduct experiments on the public dataset, the points within the provided ground truth bounding boxes are directly labeled as obstacle points.

[0146] 2) Determine the multi-scale feature data corresponding to the point cloud feature information.

[0147] Determining the multi-scale feature data includes two steps. One is to convert the point cloud feature information into pseudo-image features, and the other is to perform feature extraction operations on the pseudo-image features to obtain multi-scale features. These two steps can refer to the corresponding descriptions above.

[0148] 3) Use the multi-scale feature data and the label information to train the semantic segmentation head, the orientation information regression head, and the instance segmentation head respectively until the training stop condition is met and then stop.

[0149] In this embodiment, if the sample is annotated with a bounding box, such as a 3D bounding box, the annotated 3D bounding box in the sample data can also be mapped to the bird's-eye view through projection transformation (BEV coordinate system conversion) to supervise the training of subsequent modules.

[0150] During training, the training stop condition can be set, such as stopping when the number of training times is greater than the preset number threshold, or when the loss function value is minimized.

[0151] The entire training process can refer to Figure 10 ... The entire invention mainly includes four parts: data collection and annotation, data preprocessing design, network model design, and post-processing design. The association between each module and the data transmission process is as Figure 10 shown.

[0152] First, 2D images and Radar point cloud data are collected by the camera and the 4D imaging radar. With the assistance of the image data, the point cloud data is annotated to obtain a 3D annotation box (which can be omitted). Then, through data preprocessing operations, the Radar point cloud is columnarized and the height information is compressed, thereby converting the original point cloud information into pseudo-image feature data. At the same time, the annotated 3D annotation box is mapped to the bird's-eye view through projection transformation (BEV coordinate system conversion) to supervise the training of subsequent modules. Then, the pseudo-image feature data is input into the U-Net model for feature extraction, and the extracted features are sent to the detection head to obtain prediction results such as obstacle category, orientation, and instance discrimination. Finally, through the output results of the detection head, different-sized bounding boxes Box are predicted for different categories. The detection confidence, that is, the above-mentioned score, is calculated by judging the number of Pillars containing the obstacle category in the bounding box, and then the position and size information of the final obstacle bounding box Box are obtained through non-maximum suppression (NMS). Subsequently, it can be input into the path planning and control algorithm for use.

[0153] It should be noted that the above-mentioned bounding box is a traditional method for representing obstacles and is widely used in various deep learning-based perception tasks. However, in fact, as an optimization target and an obstacle representation method for deep learning, the 3D bounding box is not an optimal choice. According to the design idea of this algorithm, the accurate size of the bounding box is not pursued in the final output. However, in traditional detection algorithms, the mean Average Precision (mAP) is usually used as the evaluation criterion for detection performance, and the calculation of this metric needs to be based on the 3D bounding box size. Therefore, in order to compare the performance with other detection algorithms and for visualization, this post-processing design still needs to be added.

[0154] To achieve the above purpose, it is necessary to generate the corresponding 3D bounding box according to the final obstacle detection result in this algorithm, that is, to frame the detection box for the segmented result.

[0155] According to the characteristics of the 4D imaging radar itself (penetration, reflection generated in the form of waves, etc.), most of the point clouds generated are not the edge points of the target, which may affect the accuracy of the post-processing module to a certain extent. However, as mentioned above, in the actual application scenario, the accurate prediction of the 3D bounding box of the object is not pursued, and the obstacle detection result in this algorithm is mainly based on Pillar classification. Therefore, this factor will not have a great impact on the overall performance of the algorithm. Based on the starting point of visualization and performance comparison, it can be verified through experiments later which combinations of situations can balance the time complexity and accuracy requirements.

[0156] After the detection head is trained, the detection head can be used for obstacle detection operations. Specifically, refer to Figure 10 , specifically, use the 4D imaging radar to collect point cloud data, perform data preprocessing operations to obtain pseudo-image features, and then through a feature extraction network, such as a U-Net model, to extract features to obtain multi-scale features. The multi-scale features are respectively input into the obstacle segmentation network (i.e., the above-mentioned semantic segmentation head), the obstacle orientation network (i.e., the above-mentioned orientation information regression head), and the obstacle instance discrimination network (i.e., the above-mentioned instance segmentation head) to obtain category, orientation, and instance information respectively, and then fuse them. After that, perform the detection of the bounding box through the minBox post-processing model to obtain the final obstacle detection result.

[0157] In this embodiment, by using the segmentation task, the detection task is changed from the mode of framing obstacles according to contour points to judging whether the point cluster is an obstacle, which is more in line with the actual application scenario and, to a certain extent, avoids the situation of missed detection / false detection caused by the overly sparse points on the obstacles collected by the 4D imaging radar.

[0158] In addition, the accuracy of data annotation is improved. During the annotation process, it is only necessary to annotate whether the point cloud belongs to a certain type of obstacle, and there is no need to determine the complete shape of its 3D box through imagination.

[0159] Optionally, based on the embodiment of the above obstacle detection method, another embodiment of the present invention provides an obstacle detection device. Referring to Figure 11 , it may include:

[0160] A feature determination module 11, configured to acquire point cloud data collected by a millimeter-wave radar and determine the point cloud feature information of the point cloud data;

[0161] A feature processing module 12, configured to convert the point cloud feature information into pseudo-image features;

[0162] A feature extraction module 13, configured to perform a feature extraction operation on the pseudo-image features to obtain multi-scale features;

[0163] An obstacle detection module 14, configured to perform an obstacle recognition operation based on the multi-scale features to obtain the obstacle detection result of the point cloud data.

[0164] Further, the feature determination module 11 includes:

[0165] A coordinate calculation sub-module, configured to calculate the position coordinates of each point in the point cloud data;

[0166] A first feature determination sub-module, configured to determine the attribute features of each point, where the attribute features include relative speed, radar cross-section, Doppler information, and echo intensity;

[0167] A second feature determination sub-module, configured to use the position coordinates and the attribute features as the point cloud feature information.

[0168] Further, the feature processing module 12 is specifically configured to:

[0169] Perform a columnar operation on the point cloud feature information to compress the height information to obtain pseudo-image features; the pseudo-image features include multiple pillars.

[0170] Further, the feature extraction module 13 is specifically configured to:

[0171] Call a U-Net model to enable the U-Net model to perform a feature extraction operation on the pseudo-image features to obtain multi-scale features.

[0172] Further, the obstacle detection module 14 includes:

[0173] A first detection sub-module, configured to input the multi-scale features into a semantic segmentation head to perform an obstacle category detection operation to obtain an obstacle category detection result;

[0174] A second detection sub-module, configured to input the multi-scale features into an orientation information regression head for performing an orientation information detection operation to obtain an obstacle orientation information detection result;

[0175] A third detection sub-module, configured to input the multi-scale features into an instance segmentation head for performing an instance detection operation to obtain an instance classification result;

[0176] A result fusion sub-module, configured to fuse the obstacle category detection result, the obstacle orientation information detection result, and the instance classification result to obtain an obstacle detection result.

[0177] Furthermore, the semantic segmentation head, the orientation information regression head, and the instance segmentation head are trained based on sample data;

[0178] It further includes: a model training module, and the model training module includes:

[0179] A sample acquisition sub-module, configured to acquire sample data, where the sample data includes a point cloud data sample configured with point cloud feature information and label information; the label information is an obstacle category label used for training the semantic segmentation head, an orientation label used for training the orientation information regression head, and an instance label used for training the instance segmentation head;

[0180] A multi-scale data processing sub-module, configured to determine multi-scale feature data corresponding to the point cloud feature information;

[0181] A training sub-module, configured to use the multi-scale feature data and the label information to train the semantic segmentation head, the orientation information regression head, and the instance segmentation head respectively until the training stop condition is met and then stop.

[0182] Furthermore, it further includes:

[0183] A bounding box processing module, configured to determine a detection box of a strut in the obstacle detection result, count the number of struts located in the detection box, determine a score of the detection box based on the number, and determine a bounding box of each obstacle in the obstacle detection result from the detection box based on the score of the detection box.

[0184] Furthermore, when the bounding box processing module is configured to determine a detection box of a strut in the obstacle detection result, it is specifically configured to:

[0185] Obtain a detection box generation rule, where the detection box generation rule includes the relative position relationship between the strut and the detection box, and generate a detection box of each strut in the obstacle detection result according to the detection box generation rule.

[0186] Further, when the bounding box processing module is used to determine the bounding boxes of the obstacles in the obstacle detection result from the detection boxes based on the scores of the detection boxes, it is specifically used for:

[0187] Using the scores, perform non-maximum suppression operations on the detection boxes to obtain the bounding boxes of the obstacles in the obstacle detection result.

[0188] In this embodiment, when performing obstacle detection, the data collected is the point cloud data collected by a millimeter-wave radar. The millimeter-wave radar has strong penetration ability for fog, smoke, and dust, is less affected by environmental changes and extreme weather, and has strong anti-interference ability. Therefore, when using the point cloud data collected by the millimeter-wave radar for obstacle detection, the detection accuracy can be improved. In addition, in the present invention, after converting the point cloud feature information into pseudo-image features, further feature extraction operations are performed on the pseudo-image features, so that the obtained multi-scale features are richer and more representative of the features of obstacles, which will also improve the accuracy of subsequent obstacle recognition.

[0189] It should be noted that for the working processes of the various modules and sub-modules in this embodiment, please refer to the corresponding descriptions in the above embodiments, and will not be elaborated here.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An obstacle detection method, characterized in that, Including: Obtaining point cloud data collected by a millimeter-wave radar and determining point cloud feature information of the point cloud data; Converting the point cloud feature information into pseudo-image features; Performing a feature extraction operation on the pseudo-image features to obtain multi-scale features; Performing an obstacle recognition operation based on the multi-scale features to obtain an obstacle detection result of the point cloud data.

2. The obstacle detection method according to claim 1, characterized in that, Determining the point cloud feature information of the point cloud data includes: Calculating the position coordinates of each point in the point cloud data; Determining the attribute features of each point, where the attribute features include relative velocity, radar cross section, Doppler information, and echo intensity; Using the position coordinates and the attribute features as the point cloud feature information.

3. The obstacle detection method according to claim 1, characterized in that, Converting the point cloud feature information into pseudo-image features includes: Performing a columnarization operation on the point cloud feature information to compress the height information to obtain pseudo-image features; the pseudo-image features include multiple pillars.

4. The obstacle detection method according to claim 1, characterized in that, Performing a feature extraction operation on the pseudo-image features to obtain multi-scale features includes: Invoking a U-Net model to enable the U-Net model to perform a feature extraction operation on the pseudo-image features to obtain multi-scale features.

5. The obstacle detection method according to claim 1, characterized in that, Performing an obstacle recognition operation based on the multi-scale features to obtain an obstacle detection result of the point cloud data includes: Inputting the multi-scale features into a semantic segmentation head to perform an obstacle category detection operation to obtain an obstacle category detection result; Inputting the multi-scale features into an orientation information regression head to perform an orientation information detection operation to obtain an obstacle orientation information detection result; Inputting the multi-scale features into an instance segmentation head to perform an instance detection operation to obtain an instance classification result; Fusing the obstacle category detection result, the obstacle orientation information detection result, and the instance classification result to obtain an obstacle detection result.

6. The obstacle detection method according to claim 5, characterized in that, The semantic segmentation head, the orientation information regression head, and the instance segmentation head are trained based on sample data; The training process of the semantic segmentation head, the orientation information regression head, and the instance segmentation head includes: Obtaining sample data, where the sample data includes a point cloud data sample configured with point cloud feature information and label information; the label information is an obstacle category label used for training the semantic segmentation head, an orientation label used for training the orientation information regression head, and an instance label used for training the instance segmentation head; Determining multi-scale feature data corresponding to the point cloud feature information; Using the multi-scale feature data and the label information to train the semantic segmentation head, the orientation information regression head, and the instance segmentation head respectively until the training stop condition is met and then stopping.

7. The obstacle detection method according to claim 1, characterized in that, After performing an obstacle recognition operation based on the multi-scale features to obtain an obstacle detection result of the point cloud data, it further includes: Determining a detection frame of the pillars in the obstacle detection result; Counting the number of pillars located in the detection frame and determining the score of the detection frame based on the number; Based on the score of the detection frame, determining the bounding boxes of each obstacle in the obstacle detection result from the detection frame.

8. The obstacle detection method according to claim 7, characterized in that, Determining a detection frame of the pillars in the obstacle detection result includes: Obtain the detection box generation rule, where the detection box generation rule includes the relative position relationship between the struts and the detection box; Generate the detection boxes of the respective struts in the obstacle detection result according to the detection box generation rule.

9. The obstacle detection method according to claim 7, characterized in that,Based on the scores of the detection boxes, determine the bounding boxes of the respective obstacles in the obstacle detection result, including: Use the scores to perform non-maximum suppression operations on the detection boxes to obtain the bounding boxes of the respective obstacles in the obstacle detection result.

10. An obstacle detection device, characterized in that, Include: A feature determination module for obtaining the point cloud data collected by the millimeter-wave radar and determining the point cloud feature information of the point cloud data; A feature processing module for converting the point cloud feature information into pseudo-image features; A feature extraction module for performing feature extraction operations on the pseudo-image features to obtain multi-scale features; An obstacle detection module for performing obstacle recognition operations based on the multi-scale features to obtain the obstacle detection result of the point cloud data.

Images