Method, device and storage medium for predicting target speed based on millimeter-wave radar

Abstract

The present disclosure provides a method, apparatus, and storage medium for predicting the speed of a target based on a millimeter-wave radar, relating to the field of computer technologies, and particularly to fields such as image recognition, target detection, and autonomous driving. The specific implementation solution is as follows: target detection is performed based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target, and the multiple frames of millimeter-wave radar point clouds include multiple points; an initial speed of the target is obtained based on one or more points within the bounding box of the target; and a predicted speed of the target is obtained based on the initial speed of the target. In the embodiments of the present disclosure, based on multiple frames of millimeter-wave radar point clouds, the density of the millimeter-wave radar point clouds can be increased, the obtained bounding box of the target is more accurate, and a more accurate initial speed can be obtained, thereby improving the accuracy of the predicted speed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to fields such as image recognition, object detection, and autonomous driving. Background Art

[0002] Sensors such as lidar and cameras usually obtain the position information of obstacles through point clouds or neural networks, and then determine the speed of the obstacles through tracking algorithms. Therefore, the accuracy of the speed of obstacles is closely related to the tracking effect. Millimeter-wave radars can output the Doppler speed of obstacles, but traditional millimeter-wave radars have low angular resolution and are difficult to distinguish multiple obstacles that are close in distance, which easily causes false detections. Summary of the Invention

[0003] The present disclosure provides a method, an apparatus, and a storage medium for predicting the speed of an object based on a millimeter-wave radar.

[0004] According to one aspect of the present disclosure, there is provided a method for predicting the speed of an object based on a millimeter-wave radar, including:

[0005] Performing object detection based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the object, where the multiple frames of millimeter-wave radar point clouds include multiple points;

[0006] Obtaining an initial speed of the object according to one or more points within the bounding box of the object;

[0007] Obtaining a predicted speed of the object according to the initial speed of the object.

[0008] According to another aspect of the present disclosure, there is provided an apparatus for predicting the speed of an object based on a millimeter-wave radar, including:

[0009] A detection module configured to perform object detection based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the object, where the multiple frames of millimeter-wave radar point clouds include multiple points;

[0010] An acquisition module configured to obtain an initial speed of the object according to one or more points within the bounding box of the object;

[0011] A prediction module configured to obtain a predicted speed of the object according to the initial speed of the object.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method in the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program which, when executed by a processor, implements any method in the embodiments of the present disclosure.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0019] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0020] Figure 1 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to an embodiment of the present disclosure;

[0021] Figure 2 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0022] Figure 3 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0023] Figure 4 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0024] Figure 5 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0025] Figure 6 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure

[0026] Figure 7 is a flowchart of a target speed estimation technique based on a 4D millimeter-wave radar;

[0027] Figure 8Schematic block diagram of a target speed prediction device based on a millimeter-wave radar according to an embodiment of the present disclosure;

[0028] Figure 9 Schematic flowchart of a target speed prediction device based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0029] Figure 10 Block diagram of an electronic device for implementing the target speed prediction device according to an embodiment of the present disclosure. Detailed implementation manners

[0030] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted for clarity and conciseness.

[0031] Figure 1 Schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to an embodiment of the present disclosure. The method may include:

[0032] S110. Perform target detection based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target. The multiple frames of millimeter-wave radar point clouds include multiple points;

[0033] S120. Obtain an initial speed of the target based on one or more points within the bounding box of the target;

[0034] S130. Obtain a predicted speed of the target based on the initial speed of the target.

[0035] In an embodiment of the present disclosure, a millimeter-wave radar can transmit millimeter-wave signals. By measuring the reflection time, intensity, etc. of the millimeter-wave signals, information such as the position, speed, angle, height, etc. of a target, such as an obstacle, can be obtained. Different from lidar, the data structure of the millimeter-wave radar can include data structures such as RadarPoint, RadarPointCloud, and RadarFrame. For example, by using the millimeter-wave radar to scan different angles and / or directions, a millimeter-wave radar point cloud can be generated. The millimeter-wave radar point cloud collected by the radar can include information such as the position, shape, relative speed (such as Doppler speed) of the target. After the radar transmits millimeter-wave signals at a certain time point or during a certain time period, a frame of millimeter-wave radar point cloud corresponding to this certain time point or this certain time period can be measured. Multiple frames of millimeter-wave radar point cloud can be millimeter-wave radar point clouds at different time points or different time periods. Multiple frames of millimeter-wave radar point cloud (which can be simply referred to as multiple frames of point cloud) can include consecutive multiple frames of point cloud. For example, the nth frame of point cloud, the (n + 1)th frame of point cloud, and the (n + 2)th frame of point cloud. Multiple frames of millimeter-wave radar point cloud can also include non-consecutive or partially consecutive multiple frames of point cloud. For example, the nth frame of point cloud, the (n + 2)th frame of point cloud, and the (n + 3)th frame of point cloud. Multiple frames of point cloud can be spliced by multiple single-frame point clouds. For example, one frame of point cloud includes 1000 points, 4 frames of point cloud can include 4000 points, and 5 frames of point cloud can include 5000 points. The points in multiple frames of point cloud are usually denser than the points in a single-frame point cloud. Using multiple frames of point cloud for target detection can improve the accuracy of target detection.

[0036] Based on multiple frames of point cloud, a bounding box of one or more targets can be obtained. According to information such as the position and size of the bounding box of a target, usually multiple points belonging to the target, that is, points located within the bounding box of the target, can be obtained. For example, the bounding box of target A has multiple vertices. According to the coordinates of these multiple vertices, the coordinates of the points located within the bounding box of target A can be determined from multiple frames of point cloud. If the bounding box of target A is a three-dimensional bounding box, it can have 8 vertices. The x coordinate of the points within the bounding box of target A can be between the minimum x coordinate and the maximum x coordinate of these 8 vertices, the y coordinate can be between the minimum y coordinate and the maximum y coordinate of these 8 vertices, and the z coordinate can be between the minimum z coordinate and the maximum z coordinate of these 8 vertices.

[0037] In an embodiment of the present disclosure, if a bounding box of a certain target includes a point, the speed of this one point can be used as the initial speed of the target. If the bounding box of a certain target includes multiple points, the initial speed of the target can be obtained after processing the speeds of these multiple points. In an embodiment of the present disclosure, based on multiple frames of millimeter-wave radar point cloud, the density of the millimeter-wave radar point cloud can be improved, the obtained bounding box of the target is more accurate, and a more accurate initial speed can be obtained, thereby improving the accuracy of the prediction speed.

[0038] In one embodiment, the point cloud output by the millimeter-wave radar includes a four-dimensional (4D) millimeter-wave radar point cloud. The data of each point in the 4D millimeter-wave radar point cloud includes the Doppler velocity, horizontal angle, pitch angle, and three-dimensional position information of the target. Among them, the pitch angle of the target is related to the height of the target, and there is a conversion relationship between the two. Therefore, the pitch angle can be regarded as a kind of height information. In addition, the pitch angle of the target in the 4D millimeter-wave radar point cloud can also be replaced by the height of the target. Based on the height-related information in the 4D millimeter-wave radar point cloud, the three-dimensional coordinate position of the target can be obtained, reducing false detections caused by some obstacles with height, such as viaducts, speed bumps, etc.

[0039] Figure 2 FIG. is a schematic flowchart of a method for predicting the target speed based on a millimeter-wave radar according to another embodiment of the present disclosure. This method may include one or more features of the above method. In one embodiment, the method further includes:

[0040] S210. Obtain the absolute radial velocity of a point according to the Doppler velocity of the point and the moving speed of the device where the radar is located.

[0041] In the embodiment of the present disclosure, the Doppler velocity of the target in a point of the millimeter-wave radar point cloud may be the relative velocity between the device where the radar is located and the target. There are various devices where the radar is located. For example, ordinary vehicles, autonomous driving vehicles, aircraft, etc. According to the electromagnetic wave signal transmitted by the radar on the radar vehicle, such as a millimeter-wave signal, and the phase, frequency, etc. of the echo signal after the millimeter-wave signal reaches the target, the Doppler velocity of the target can be calculated. According to the Doppler velocity of the target in a point, the absolute radial velocity of the target in this point can be calculated. The absolute radial velocity can distinguish static and dynamic obstacles. For example, the absolute radial velocity of a static obstacle is usually 0, and the absolute radial velocity of a dynamic obstacle is usually not 0 or not 0 for a long time. The absolute radial velocity can be understood as the velocity component in the direction of the line connecting the radar and the obstacle.

[0042] In the embodiment of the present disclosure, obtaining the absolute radial velocity according to the Doppler velocity of a point is beneficial to distinguishing the types of obstacles, reducing the interference of static obstacles, improving the processing efficiency, and further improving the accuracy of speed estimation and / or tracking of dynamic obstacles.

[0043] In the embodiments of the present disclosure, before calculating the absolute radial velocity of a target in a point of millimeter-wave radar point cloud, some preprocessing can be performed on the millimeter-wave radar point cloud. For example, the preprocessing can include filtering, which can include one or more of the following: filtering out points with abnormal values, filtering out points with a relatively large range, or filtering out points with too high a height. Among them, points with abnormal values can include points with nan (not a number) values. Points with a relatively large range can include points with values exceeding the normal range, such as 1000 meters. Points with too high a height can include points with a height greater than a certain threshold. Again, the preprocessing can include reflection intensity correction. For example, the reflection velocity can be corrected using the formula y = ax + b. Where x is the reflection intensity returned by the radar, y is the corrected reflection intensity, and a and b can be user-defined values. Correcting the reflection intensities of points of different models of millimeter-wave radars can reduce the differences in reflection intensities of different models of radars.

[0044] In the embodiments of the present disclosure, after filtering, correcting the reflection intensity, and calculating the absolute radial velocity, the absolute radial velocity of each point can be associated with other features of the point after preprocessing and participate in subsequent target detection. In this way, the accuracy of the target detection result can be improved.

[0045] In one implementation, step S210 obtains the absolute radial velocity of a point based on the Doppler velocity of the point and the motion velocity of the device where the radar is located, including:

[0046] S220. Calculate the absolute radial velocity of the point based on the motion velocity, rotation matrix, and the Doppler velocity, horizontal angle, and pitch angle of the point; where the rotation matrix is the rotation matrix from the coordinate system of the device where the radar is located to the millimeter-wave radar coordinate system.

[0047] In the embodiments of the present disclosure, the conversion relationship between the Cartesian coordinate system and the polar coordinate system can be used, combined with the Doppler velocity of the target in a point and the velocity of the device where the radar is located, etc., to calculate the absolute radial velocity of the target in the point. Usually, the data returned by the radar is in the polar coordinate system, and the coordinate systems of the device where the radar is located (such as the host vehicle) and the millimeter-wave coordinate system are in the Cartesian coordinate system. The data returned by the radar can be converted from the polar coordinate system to the Cartesian coordinate system, and the host vehicle velocity can be converted from the host vehicle coordinate system (Cartesian coordinate system) to the millimeter-wave coordinate system (Cartesian coordinate system). Then, the data returned by the radar in the Cartesian coordinate system and the host vehicle velocity are added to obtain the absolute radial velocity.

[0048] The following is an example of a formula for calculating the absolute radial velocity:

[0049] v car_r = v car * R

[0050] v comp = v + (vcar_r [0]*cos(α)+v car_r [1]*sin(α)+v car_r [2]*sin(θ))

[0051] where v car can represent the speed of the host vehicle (an example of the device where the radar is located) output by the positioning device, and can be a three-dimensional vector; R represents the rotation matrix from the host vehicle coordinate system to the 4D millimeter wave coordinate system; v car_r is also a three-dimensional vector, v car_r [0], v car_r [1], v car_r [2] where 1, 2, and 3 represent the indices of the vector dimensions; v represents the Doppler speed of the target measured by the 4D millimeter wave radar and can be a scalar; α represents the horizontal angle of the target measured by the millimeter wave radar; θ represents the pitch angle of the target measured by the millimeter wave radar; v comp represents the absolute radial velocity (which can be simply referred to as the absolute velocity) of each point in the final output.

[0052] In the embodiments of the present disclosure, according to the movement speed and rotation matrix of the device where the radar is located, as well as the Doppler speed, horizontal angle, and pitch angle of a point, the absolute radial velocity of this point can be calculated quickly and accurately.

[0053] Figure 3 is a schematic flowchart of a method for predicting the target speed based on a millimeter wave radar according to another embodiment of the present disclosure. This method may include one or more features of the above method. In one implementation, step S110 includes:

[0054] S310. Extract point features from the multi-frame millimeter wave radar point cloud;

[0055] S320. Input the point features into a point cloud target detection model to output target features;

[0056] S330. Determine the bounding box of the target according to the target features.

[0057] In the embodiments of the present disclosure, the point cloud target detection model can be pre-trained. The point cloud target detection model can include artificial intelligence models such as neural network models. The point cloud target detection model can identify the features of multiple points in the input millimeter wave radar point cloud to obtain the features of one or more targets. According to the features of each target, the bounding box of the target can be determined, such as the 8 vertices of the bounding box, and then the points belonging to each bounding box can be determined. The points in the bounding box of a certain target can be considered as the points belonging to this target. Based on the point cloud target detection model, the accuracy of the bounding box of the recognized target can be improved, and thus the accurate points belonging to the target can be obtained.

[0058] In one implementation, the point features input to the point cloud target detection model include the three-dimensional position information, reflection intensity, Doppler velocity, and absolute radial velocity of the points, and the target features output by the point cloud target detection model include at least one of the three-dimensional position information, three-dimensional bounding box size, and three-dimensional bounding box heading angle of the target. For example, a multi-frame millimeter-wave radar point cloud includes 3,000 points. The three-dimensional position information, reflection intensity, Doppler velocity, and absolute radial velocity of these 3,000 points can be extracted respectively first, and then can be input into the point cloud target detection model in one batch or in batches. The point cloud target detection model can output features such as the three-dimensional position information, three-dimensional bounding box size, and three-dimensional bounding box heading angle of one or more targets according to the input features. For example, the three-dimensional position information of the target can include the three-dimensional coordinates of the center point, end point, etc. of the target. Another example is that the three-dimensional bounding box size of the target can include information such as length, width, and height. The three-dimensional bounding box heading angle can be regarded as the heading angle of the target. Features such as the three-dimensional position information, three-dimensional bounding box size, and three-dimensional bounding box heading angle of the target can be used to determine the three-dimensional coordinates of the 8 vertices of the three-dimensional bounding box. Furthermore, the three-dimensional coordinates of each point in the multi-frame millimeter-wave radar point cloud that belong to the respective bounding boxes can be determined. For example, points p1, p2, and p5 belong to bounding box BBox1, and points p3, p4, p6, and p7 belong to bounding box BBox2. The number of points in the bounding box in the example is small. In actual application scenarios, there may be many points in one bounding box, such as a dozen, dozens, hundreds, etc. By using the trained point cloud target detection model, based on the target features identified from the point features in the input point cloud, since the input point features can include not only the three-dimensional position information, reflection intensity, and Doppler velocity of the points, but also the absolute radial velocity of the points, the identified target features are more accurate.

[0059] In one implementation, the training samples of the point cloud object detection model include: millimeter-wave radar point cloud samples and 3D annotation data; wherein, the 3D annotation data includes at least one of the category of the object, whether the object is truncated, whether the object is occluded, the observation azimuth angle, the vertex coordinates of the 2D bounding box, the length, width and height of the 3D bounding box, the center point of the 3D bounding box, and the orientation angle of the 3D bounding box. For example, during the model training process, the 3D position information, reflection intensity, Doppler velocity, absolute radial velocity and other point features of each point can be extracted from the 4D millimeter-wave radar point cloud samples first. The extracted point features are input into a point cloud object detection model, such as a neural network-based object model. The model outputs object features such as the 3D position information of the detected obstacle, the length, width and height of the 3D bounding box (3D BBox), and the orientation angle of the 3D BBox. Then, the loss function is calculated based on the output object features and the 3D annotation data, and the parameters of the model are updated according to the calculation result of the loss function. The above steps are executed multiple times until the loss function converges, and a trained point cloud object detection model is obtained. The model structure and loss function used for training can be selected according to actual requirements such as accuracy and computing power, and the embodiments of the present disclosure do not make any limitations. Using the training samples composed of millimeter-wave radar point cloud samples and 3D annotation data to train a point cloud object detection model can detect more accurate object features based on the millimeter-wave radar point cloud.

[0060] In one implementation, the training samples of the point cloud object detection model further include the external parameters of the millimeter-wave radar, and the external parameters of the millimeter-wave radar include the conversion parameters from the millimeter-wave radar to the positioning device and the pose of the vehicle where the radar is located output by the positioning device. The positioning device may include, for example, the hardware required for positioning systems such as GPS and Beidou navigation systems. The positioning device and the radar may be in the same device, for example, both in the host vehicle. The external parameters of the millimeter-wave radar can be used to calculate the absolute radial velocity, increase the point features in the millimeter-wave radar point cloud, and improve the accuracy of subsequent object detection results.

[0061] Figure 4 It is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure. This method may include one or more features of the above method. In one implementation, this method further includes:

[0062] S410. Stitch the current frame point cloud and the previous N frame point clouds of the current frame point cloud to obtain a stitched point cloud; where N is greater than or equal to 1. In the embodiments of the present disclosure, multiple frame point clouds can be stitched to obtain a stitched point cloud. For example, the first 4 frame point clouds include 4000 points, the current frame point cloud includes 1000 points, and the stitched 5 frame point clouds include 5000 points. The stitched point cloud has better density than a single frame point cloud, and can improve the accuracy of subsequent object detection results using the stitched point cloud.

[0063] In one embodiment, as Figure 4 shown, step S410 includes: converting the first N frames of point clouds from the millimeter-wave radar coordinate system to the world coordinate system and storing them in a queue; when the current frame of point cloud is acquired, reading the first N frames of point clouds from the queue, and converting the pose data of the first N frames of point clouds from the world coordinate system to the millimeter-wave radar coordinate system; splicing the converted point clouds with the current frame of point cloud acquired by the millimeter-wave radar to obtain the spliced point cloud. In the embodiments of the present disclosure, the first N frames of point clouds can be understood as the historical frame point clouds acquired before the current frame. The historical frame point clouds can be stored through a data structure such as a queue. The queue can store the frame identifier, point identifier, point data, etc. of the historical frame point clouds. The point data can include the pose data of one or more of the horizontal angle, pitch angle, three-dimensional coordinate position, reflection intensity, Doppler velocity, and absolute radial velocity of the target in the point. Generally, the data returned by the millimeter-wave radar can be converted to the millimeter-wave coordinate system. After preprocessing these point clouds and calculating the absolute radial velocity, they can be converted to the world coordinate system and saved to the queue first. Only the first N frames of point clouds can be saved in the queue, or more frames of point clouds can be saved. The number of point cloud frames to be taken out can be adjusted according to the actual application scenario. When the millimeter-wave radar acquires the current frame of point cloud, a read command can be triggered to read the historical frame point clouds in the world coordinate system from the queue. Then, the historical frame point clouds are converted to the millimeter-wave coordinate system and spliced with the current frame of point cloud (which can be preprocessed and the absolute radial velocity can be calculated) to obtain the spliced point cloud.

[0064] Using the queue to save the historical first N frames of point clouds in advance facilitates quickly obtaining the spliced point cloud after the current frame of point cloud is acquired, improves the calculation speed, and further uses the denser spliced point cloud for target detection later, improving the accuracy of the target detection result.

[0065] In one embodiment, step S310 may further include:

[0066] S420. Extracting the point features from the spliced point cloud. In one way, the point features can be extracted from the spliced point cloud first, and then the point features extracted from the spliced point cloud can be input into the point cloud target detection model. In another way, the point cloud target detection model can also have the function of extracting point features from the point cloud, which is equivalent to including a feature extraction layer in the point cloud target model. In this case, the spliced point cloud can be input into the point cloud target detection model, and the point cloud target detection model extracts the point features and then continues the subsequent target detection process. Using the spliced point cloud, a more accurate target detection result can be obtained.

[0067] Figure 5 is a schematic flowchart of a target speed prediction method based on a millimeter-wave radar according to another embodiment of the present disclosure. This method may include one or more features of the above method. In one embodiment, step S120 includes:

[0068] S510. Calculate the average velocity based on the absolute radial velocities of multiple points within the bounding box of the target. This average velocity serves as the initial velocity of the target. By taking the average of the absolute radial velocities of multiple points belonging to the target within the target's bounding box, the resulting average velocity can reduce single-point errors and be closer to the actual absolute radial velocity of the target. Using this average velocity as the initial velocity of the target for subsequent prediction is conducive to obtaining a more accurate predicted velocity.

[0069] In one implementation, step S130 includes:

[0070] S520. Use the initial velocity of the target as the starting state of the Kalman filter algorithm, and use the Kalman filter algorithm to output the predicted velocity of the target. In the embodiments of the present disclosure, the initial velocity of the target can be used as the initial velocity at a certain moment, such as moment t, and the Kalman filter algorithm is used to predict the predicted velocity of the target at the next moment, such as moment t + 1. Then, based on moment t + 1, the Kalman filter algorithm is used again to predict the predicted velocity of the target at moment t + 2. And so on, thereby realizing the prediction of the velocities of the target at subsequent different moments. Prediction based on the average velocity of the absolute radial velocity results in a more accurate predicted velocity.

[0071] Furthermore, if the velocity of the target is correlated with other states of the target, such as pitch angle, altitude, horizontal angle, three-dimensional position information, etc., predictions of other states of the target can also be made.

[0072] Figure 6 FIG. is a schematic flowchart of a method for predicting the velocity of a target based on a millimeter-wave radar according to another embodiment of the present disclosure. This method may include one or more features of the above method. In one implementation, the method further includes:

[0073] S610. Convert the center position of the target included in the bounding box from millimeter-wave radar coordinates to world coordinates;

[0074] S620. Filter out the targets whose center positions are not within the region of interest of the map.

[0075] In an embodiment of the present disclosure, the coordinates of the center position of the three-dimensional bounding box of the target can be converted, from the coordinates of the millimeter-wave radar to the world coordinates. Then, based on the converted center position, a comparison can be made with the region of interest in the world coordinate system on the map. The region of interest on the map may include a region within a certain range in front of the vehicle driving position. For example, the region of interest on the map includes a rectangular or cuboid region within 200 meters in front of the vehicle or within the range of machine vision. According to the vertices of the rectangular or cuboid region, it can be determined whether the center position of the three-dimensional bounding box of the target is within the region of interest. If the center position of the three-dimensional bounding box of the target is within the region of interest, the data of the target is retained for subsequent tracking or speed prediction. If the center position of the three-dimensional bounding box of the target is outside the region of interest, the data of the target can be discarded and no subsequent tracking or speed prediction is performed. Filtering out the targets whose center positions are not within the region of interest on the map can reduce the tracking and speed prediction of unnecessary targets, reduce the computational workload of the device, and improve the efficiency of tracking and speed prediction of necessary targets.

[0076] In one implementation, the method further includes:

[0077] S630. Using the Hungarian matching algorithm, match and track the target according to at least one of the position, size, category, heading, and bounding box of the target. In an embodiment of the present disclosure, the Hungarian matching algorithm can be used to match and track the retained targets after filtering. For example, the Hungarian algorithm can be used to establish a graph, where there are targets in two adjacent frames. Then calculate the mutual distance between the nodes of the two frames. The smaller this distance, the greater the probability that the two frames include the same target. Using the Hungarian algorithm, the matching and tracking of the target can be achieved based on the position, size, category, heading, and bounding box of the target. Step S630 is combined with steps S510, S520. For example, according to the tracking result of S630, determine the bounding boxes of the same target at different times, and then calculate the average radial velocity of the target at a certain time based on the points in the bounding box of the target at that time, so as to obtain the predicted velocity.

[0078] Since traditional millimeter-wave radars cannot measure the target height information, it often causes false detections for viaducts, speed bumps, etc., and it is difficult to distinguish various static obstacles. Moreover, the angular resolution of traditional millimeter-wave radars is very low, and they cannot distinguish multiple obstacles that are close in distance and cannot obtain the shape information of the obstacles.

[0079] Compared with traditional millimeter-wave radars, 4D millimeter-wave radars have added elevation measurement and can output data such as the horizontal angle, altitude angle (e.g., pitch angle), and distance of a target, from which the x, y, and z of the target, i.e., three-dimensional position information, can be calculated. Moreover, the angular resolution of 4D millimeter-wave radars has been significantly improved, and methods such as deep learning can be used to extract point features and identify the semantic information of the target, avoiding false detections caused by noise in traditional millimeter-wave radars. The method for predicting the target speed based on a millimeter-wave radar proposed in the embodiments of the present disclosure may include a method for estimating the target speed of a 4D millimeter-wave radar based on deep learning, which can avoid the speed measurement problems existing in sensors such as the above-mentioned traditional millimeter-wave radars, lidars, and cameras, and improve the speed estimation accuracy of a vehicle, such as an autonomous vehicle, for a target, such as an obstacle.

[0080] In the embodiments of the present disclosure, the device where the radar is located includes, but is not limited to, a vehicle equipped with a 4D millimeter-wave radar, and may also include other movable devices with 4D millimeter-wave radars.

[0081] An example of a method for estimating the target speed based on a 4D millimeter-wave radar mainly includes the following stages, as Figure 7 shown.

[0082] S710. The 4D millimeter-wave radar inputs point clouds.

[0083] S720. Point cloud preprocessing: For the point clouds output by the 4D millimeter-wave radar driving module, the point cloud preprocessing module will perform preprocessing on the point clouds. The preprocessing may include filtering, such as filtering out points with abnormal values, points at a relatively far distance, and points with a height higher than a certain threshold. The preprocessing may also include correcting the reflection intensity of each point to avoid differences in reflection intensity of different millimeter-wave models. The preprocessing may also include calculating the absolute speed. The point clouds output by the radar usually include the Doppler speed of each point. Combining the speed of the host vehicle, the absolute speed of each point, that is, the speed of each point in the radar radial direction, can be calculated, which can also be called the absolute radial speed. The following is an example of the formula for calculating the absolute radial speed:

[0084] v car_r = v car * R

[0085] v comp = v + (v car_r [0] * cos(α) + v car_r [1] * sin(α) + v car_r [2] * sin(θ))

[0086] where v carRepresents the main vehicle speed output by the positioning device, which is a three-dimensional vector. R represents the rotation matrix from the main vehicle coordinate system to the 4D millimeter-wave coordinate system. v represents the Doppler speed of the target measured by the 4D millimeter-wave radar, which is a scalar. α represents the horizontal angle of the target measured by the 4D millimeter-wave radar, and θ represents the pitch angle of the target measured by the 4D millimeter-wave radar. v comp Represents the absolute radial velocity of each point in the final output.

[0087] S730. Object detection based on deep learning (point cloud obstacle recognition): First, establish a queue that stores N frames, such as 4 frames of 4D millimeter-wave point clouds, before the current frame. After collecting the current frame of point cloud, first filter out the timed-out point clouds in the queue, and then transform the previous 4 frames of point clouds to the 4D millimeter-wave radar coordinate system according to the pose of the current frame and splice them with the current frame of point cloud to obtain 5 frames of spliced point clouds. Input the three-dimensional position information, reflection intensity, Doppler speed, and absolute radial speed of the points in the 5 frames of 4D millimeter-wave radar point clouds into the point cloud object detection model based on deep learning (deep neural network) for object detection of the 4D millimeter-wave radar.

[0088] S740. Preliminary estimation of target speed and target post-processing: Obtain the three-dimensional position, size, and heading angle of the target, such as an obstacle, output by the deep neural network, calculate the eight vertices of the obstacle, and then obtain the bounding box of the obstacle. Calculate the 4D millimeter-wave radar points of the obstacle within the bounding box according to the bounding box of the obstacle. Calculate the average value of the absolute radial speeds of the points of the obstacle within the bounding box based on the absolute radial speed of each point calculated in step (1) to obtain the preliminary speed of the obstacle. Since the millimeter-wave radar cannot measure the tangential speed, it is necessary to use the Kalman filter in combination with step S750 to correct the speed. Convert the center position of the obstacle to the world coordinate, determine whether the obstacle is within the region of interest (ROI) of the map, and filter out the obstacles outside the ROI.

[0089] S750. Target tracking: Use the Hungarian matching algorithm to match and track the above-mentioned output obstacles according to various information such as the position, size, bounding box, and category of the obstacles.

[0090] S760. Speed output: Use the preliminary speed of the obstacle obtained in step S740 as the starting state of the Kalman filter algorithm, and use the Kalman filter to output the speed of the obstacle as the final output.

[0091] Traditional target speed estimation techniques based on lidar and cameras can only calculate the speed of obstacles through the position of the obstacles and tracking algorithms. If the tracking algorithm has poor performance and there are matching errors, it will lead to incorrect speeds output by the tracking algorithm. For traditional millimeter-wave radar-based target speed estimation techniques, since traditional millimeter-waves cannot measure the target height and have low angular resolution, false detections or missed detections often occur; for the 4D millimeter-wave radar speed estimation method based on model output speed, the Doppler speed cannot be explicitly utilized, and the speed estimation accuracy is relatively low; for the 4D millimeter-wave radar speed estimation method based on clustering algorithms, due to the accuracy limitations of the clustering algorithm, the target point cloud range is incorrect, resulting in incorrect average radial speed calculations.

[0092] The solution of the embodiments of the present disclosure uses a 4D millimeter-wave radar. After point cloud preprocessing, a deep neural network is used to extract features and detect targets from the 4D millimeter-wave radar point cloud, outputting accurate information such as the position and size of the target, and then calculating the point cloud of the target and its average radial speed. Further, using the estimated average radial speed as the initial state of the Kalman filter, the absolute radial speed of the tracked target can be output.

[0093] Figure 8 It is a schematic block diagram of a target speed prediction device based on a millimeter-wave radar according to an embodiment of the present disclosure. The device 800 may include:

[0094] A detection module 801, configured to perform target detection based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target, where the multiple frames of millimeter-wave radar point clouds include multiple points;

[0095] An acquisition module 802, configured to obtain an initial speed of the target according to one or more points within the bounding box of the target;

[0096] A prediction module 803, configured to obtain a predicted speed of the target according to the initial speed of the target.

[0097] Figure 9 It is a schematic flowchart of a target speed prediction device based on a millimeter-wave radar according to another embodiment of the present disclosure. The device 900 may include: a detection module 901, an acquisition module 902, and a prediction module 903. The functions of these modules can refer to the relevant descriptions in the device 800.

[0098] In one implementation, the device further includes:

[0099] A speed calculation module 904, configured to obtain the absolute radial speed of a point according to the Doppler speed of the point and the movement speed of the device where the radar is located.

[0100] In one embodiment, the speed calculation module 904 is configured to calculate the absolute radial velocity of the point based on the motion speed, the rotation matrix, and the Doppler velocity, horizontal angle, and pitch angle of the point; wherein, the rotation matrix is the rotation matrix from the device coordinate system where the radar is located to the millimeter-wave radar coordinate system.

[0101] In one embodiment, the detection module 901 includes:

[0102] A feature extraction sub-module 9011, configured to extract point features from the multi-frame millimeter-wave radar point cloud;

[0103] A feature input sub-module 9012, configured to input the point features into a point cloud object detection model to output object features;

[0104] A determination sub-module 9013, configured to determine the bounding box of the object according to the object features.

[0105] In one embodiment, the point features input to the point cloud object detection model include the three-dimensional position information, reflection intensity, Doppler velocity, and absolute radial velocity of the point, and the object features output by the point cloud object detection model include at least one of the three-dimensional position information, three-dimensional bounding box size, and three-dimensional bounding box heading angle of the object.

[0106] In one embodiment, the training samples of the point cloud object detection model include: millimeter-wave radar point cloud samples, three-dimensional annotation data; wherein, the three-dimensional annotation data includes at least one of the category of the object, whether the object is truncated, whether the object is occluded, the observation azimuth angle, the vertex coordinates of the two-dimensional bounding box, the length, width, and height of the three-dimensional bounding box, the center point of the three-dimensional bounding box, and the orientation angle of the three-dimensional bounding box.

[0107] In one embodiment, the training samples of the point cloud object detection model further include the external parameters of the millimeter-wave radar, and the external parameters of the millimeter-wave radar include the conversion parameters from the millimeter-wave radar to the positioning device and the pose of the vehicle where the radar is located output by the positioning device.

[0108] In one embodiment, it further includes:

[0109] A splicing module 905, configured to splice the current frame point cloud and the previous N frame point clouds of the current frame point cloud to obtain a spliced point cloud; where N is greater than or equal to 1.

[0110] In one embodiment, the splicing module 905 includes:

[0111] A conversion sub-module 9051, configured to convert the previous N frame point clouds from the millimeter-wave radar coordinate system to the world coordinate system and store them in a queue;

[0112] A reading sub-module 9052, configured to read the previous N frame point clouds from the queue when the current frame point cloud is acquired, and convert the pose data of the previous N frame point clouds from the world coordinate system to the millimeter-wave radar coordinate system;

[0113] A splicing sub-module 9053, configured to splice the converted point cloud with the current frame point cloud acquired by the millimeter-wave radar to obtain the spliced point cloud.

[0114] In one implementation, the feature extraction sub-module 901 is further configured to extract the point features from the spliced point cloud.

[0115] In one implementation, the obtaining module 902 is configured to calculate an average speed based on the absolute radial speeds of multiple points within the bounding box of the target, and the average speed is the initial speed of the target.

[0116] In one implementation, the prediction module 903 is configured to use the initial speed of the target as the starting state of the Kalman filter algorithm, and output the predicted speed of the target using the Kalman filter algorithm.

[0117] In one implementation, it further includes:

[0118] A conversion module 906, configured to convert the center position of the target included in the bounding box from the millimeter-wave radar coordinate to the world coordinate;

[0119] A filtering module 907, configured to filter out the targets whose center positions are not within the region of interest of the map.

[0120] In one implementation, it further includes:

[0121] A tracking module 908, configured to use the Hungarian matching algorithm to match and track the target according to at least one of the position, size, category, heading, and bounding box of the target.

[0122] In one implementation, the point cloud output by the millimeter-wave radar includes a 4D millimeter-wave radar point cloud, and the data of each point in the 4D millimeter-wave radar point cloud includes Doppler velocity, horizontal angle, pitch angle, and three-dimensional position information.

[0123] For the specific functions and examples of the modules and sub-modules of the device in the embodiments of the present disclosure, reference may be made to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be elaborated herein.

[0124] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0125] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0126] Figure 10 FIG. shows a schematic block diagram of an exemplary electronic device 1000 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital assistant, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] As Figure 10 shown, the device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0128] A plurality of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as the target speed prediction method based on millimeter-wave radar. For example, in some embodiments, the target speed prediction method based on millimeter-wave radar can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the target speed prediction method based on millimeter-wave radar described above can be executed. Alternatively, in other embodiments, the computing unit 1001 can be configured to execute the target speed prediction method based on millimeter-wave radar in any other suitable manner (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of a communication network include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0136] It should be understood that various forms of the processes shown above may be used, steps may be reordered, added or deleted. For example, the steps recited in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitations are imposed herein.

[0137] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A target speed prediction method based on millimeter-wave radar, comprising: Performing target detection according to multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target, wherein the multiple frames of millimeter-wave radar point clouds include multiple points; Obtaining an initial speed of the target according to one or more points within the bounding box of the target; Obtaining a predicted speed of the target according to the initial speed of the target; Wherein, performing target detection according to multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target includes: Extracting point features from the multiple frames of millimeter-wave radar point clouds; Inputting the point features into a point cloud target detection model to output target features; Determining the bounding box of the target according to the target features; Wherein, the point features input to the point cloud target detection model include the three-dimensional position information of the points, reflection intensity, Doppler speed, and absolute radial speed, and the target features output by the point cloud target detection model include at least one of the three-dimensional position information of the target, three-dimensional bounding box size, and three-dimensional bounding box heading angle.

2. The method according to claim 1, further comprising: Obtaining the absolute radial speed of a point according to the Doppler speed of the point and the movement speed of the device where the radar is located.

3. The method according to claim 2, wherein, Obtaining the absolute radial speed of a point according to the Doppler speed of the point and the movement speed of the device where the radar is located, including: Calculating the absolute radial speed of the point according to the movement speed, rotation matrix, and the Doppler speed, horizontal angle, and pitch angle of the point; Wherein, the rotation matrix is the rotation matrix from the coordinate system of the device where the radar is located to the millimeter-wave radar coordinate system.

4. The method according to claim 1, wherein The training samples of the point cloud target detection model include: millimeter-wave radar point cloud samples, three-dimensional annotation data; wherein, the three-dimensional annotation data includes at least one of the category of the target, whether the target is truncated, whether the target is occluded, observation azimuth angle, vertex coordinates of the two-dimensional bounding box, length, width, and height of the three-dimensional bounding box, center point of the three-dimensional bounding box, and orientation angle of the three-dimensional bounding box.

5. The method according to claim 4, wherein, The training samples of the point cloud target detection model further include millimeter-wave radar extrinsic parameters, and the millimeter-wave radar extrinsic parameters include conversion parameters from the millimeter-wave radar to the positioning device and the pose of the vehicle where the radar is located output by the positioning device.

6. The method according to any one of claims 4 or 5, further comprising: Stitching the current frame of point cloud and the previous N frames of point clouds of the current frame of point cloud to obtain a stitched point cloud; Where N is greater than or equal to 1.

7. The method according to claim 6, stitching the current frame of point cloud and the previous N frames of point clouds of the current frame of point cloud to obtain a stitched point cloud, including: Converting the previous N frames of point clouds from the millimeter-wave radar coordinate system to the world coordinate system and storing them in a queue; When the current frame of point cloud is collected, reading the previous N frames of point clouds from the queue and converting the pose data of the previous N frames of point clouds from the world coordinate system to the millimeter-wave radar coordinate system; Stitching the converted point cloud with the current frame of point cloud collected by the millimeter-wave radar to obtain the stitched point cloud.

8. The method according to claim 6 or 7, wherein Extracting point features from the multiple frames of millimeter-wave radar point clouds, including: Extracting the point features from the stitched point cloud.

9. The method according to any one of claims 1 to 8, wherein Obtaining an initial velocity of the target based on one or more points within the bounding box of the target includes: Calculating an average velocity based on the absolute radial velocities of a plurality of points within the bounding box of the target, where the average velocity is the initial velocity of the target.

10. The method according to any one of claims 1 to 9, wherein, Obtaining a predicted velocity of the target based on the initial velocity of the target includes: Taking the initial velocity of the target as the starting state of the Kalman filter algorithm and using the Kalman filter algorithm to output the predicted velocity of the target.

11. The method according to any one of claims 1 to 10 further includes: Converting the center position of the target included in the bounding box from millimeter-wave radar coordinates to world coordinates; Filtering out targets whose center positions are not within the region of interest of the map.

12. The method according to any one of claims 1 to 11 further includes: Using the Hungarian matching algorithm to match and track the target according to at least one of the position, size, category, heading, and bounding box of the target.

13. The method according to any one of claims 1 to 12, wherein The point cloud output by the millimeter-wave radar includes a four-dimensional (4D) millimeter-wave radar point cloud, and the data of each point in the 4D millimeter-wave radar point cloud includes Doppler velocity, horizontal angle, pitch angle, and three-dimensional position information.

14. A target velocity prediction device based on a millimeter-wave radar, comprising: A detection module configured to perform target detection based on multiple frames of millimeter-wave radar point clouds to obtain a bounding box of the target, where the multiple frames of millimeter-wave radar point clouds include a plurality of points; An acquisition module configured to obtain an initial velocity of the target based on one or more points within the bounding box of the target; A prediction module configured to obtain a predicted velocity of the target based on the initial velocity of the target; Wherein, the detection module includes: A feature extraction sub-module configured to extract point features from the multiple frames of millimeter-wave radar point clouds; A feature input sub-module configured to input the point features into a point cloud target detection model to output target features; A determination sub-module configured to determine the bounding box of the target according to the target features; Wherein, the point features input into the point cloud target detection model include three-dimensional position information, reflection intensity, Doppler velocity, and absolute radial velocity of the points, and the target features output by the point cloud target detection model include at least one of three-dimensional position information, three-dimensional bounding box size, and three-dimensional bounding box heading angle of the target.

15. The device according to claim 14, the device further includes: A velocity calculation module configured to obtain the absolute radial velocity of a point according to the Doppler velocity of the point and the motion velocity of the device where the radar is located.

16. The apparatus according to claim 15, wherein, The velocity calculation module is configured to calculate the absolute radial velocity of the point according to the motion velocity, rotation matrix, and the Doppler velocity, horizontal angle, and pitch angle of the point; wherein, the rotation matrix is the rotation matrix from the coordinate system of the device where the radar is located to the millimeter-wave radar coordinate system.

17. The device according to claim 14, wherein, The training samples of the point cloud target detection model include: millimeter-wave radar point cloud samples, 3D annotation data; wherein, the 3D annotation data includes at least one of the category of the target, whether the target is truncated, whether the target is occluded, the observation azimuth angle, the vertex coordinates of the 2D bounding box, the length, width and height of the 3D bounding box, the center point of the 3D bounding box, and the orientation angle of the 3D bounding box.

18. The device according to claim 17, wherein, The training samples of the point cloud target detection model further include the external parameters of the millimeter-wave radar, and the external parameters of the millimeter-wave radar include the conversion parameters from the millimeter-wave radar to the positioning device and the pose of the vehicle where the radar is located output by the positioning device.

19. The device according to any one of claims 17 to 18, further comprising: A splicing module for splicing the current frame of point cloud and the first N frames of point clouds of the current frame of point cloud to obtain a spliced point cloud; where N is greater than or equal to 1.

20. The device according to claim 19, wherein the splicing module comprises: A conversion sub-module for converting the first N frames of point clouds from the millimeter-wave radar coordinate system to the world coordinate system and storing them in a queue; A reading sub-module for, when the current frame of point cloud is acquired, reading the first N frames of point clouds from the queue and converting the pose data of the first N frames of point clouds from the world coordinate system to the millimeter-wave radar coordinate system; A splicing sub-module for splicing the converted point cloud with the current frame of point cloud acquired by the millimeter-wave radar to obtain the spliced point cloud.

21. The apparatus according to claim 19 or 20, wherein The feature extraction sub-module is further configured to extract the point features from the spliced point cloud.

22. The device according to any one of claims 14 to 21, wherein The acquisition module is configured to calculate an average speed according to the absolute radial speeds of a plurality of points in the bounding box of the target, and the average speed is the initial speed of the target.

23. The apparatus according to any one of claims 14 to 22, wherein The prediction module is configured to use the initial speed of the target as the starting state of the Kalman filtering algorithm and output the predicted speed of the target using the Kalman filtering algorithm.

24. The device according to any one of claims 14 to 23, further comprising: A conversion module for converting the central position of the target included in the bounding box from the millimeter-wave radar coordinate to the world coordinate; A filtering module for filtering out the targets whose central positions are not within the region of interest of the map.

25. The device according to any one of claims 14 to 24, further comprising: A tracking module for using the Hungarian matching algorithm to match and track the target according to at least one of the position, size, category, heading and bounding box of the target.

26. The device according to any one of claims 14 to 25, wherein The point cloud output by the millimeter-wave radar includes a 4D millimeter-wave radar point cloud, and the data of each point in the 4D millimeter-wave radar point cloud includes Doppler velocity, horizontal angle, pitch angle and 3D position information.

27. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-13.

Images