Multi-radar based target tracking method, device, equipment and storage medium

Abstract

The disclosed embodiments relate to a target tracking method, device, equipment and storage medium based on multiple radars. The method includes: determining the target detection object information of the detection object based on multiple current frame point clouds obtained by scanning the target area by multiple radar sensors with different scanning angles; predicting the initial tracking object information of the initial tracking object corresponding to each current frame point cloud based on the tracking object information of the previous frame point cloud; matching the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result; determining the target tracking object and the target tracking object information corresponding to each current frame point cloud based on the matching result, and implementing target tracking based on the target tracking object and the target tracking object information. In this way, the problem of losing the tracking object in the case of sparse point clouds is avoided, and the persistence of target tracking is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a multi-radar based target tracking method, device, equipment and storage medium. Background Art

[0002] With the acceleration of urbanization and the increase in the number of vehicles, urban traffic problems are becoming increasingly prominent, such as traffic congestion, frequent accidents, and increased pollution. Traditional traffic management methods can no longer meet the growing demand, so the introduction of intelligent transportation systems has become one of the solutions. Intelligent transportation systems use advanced information and communication technologies, as well as artificial intelligence technologies, to improve urban traffic management efficiency, optimize traffic flow, and improve traffic safety and the environment.

[0003] In the intelligent transportation system, target tracking technology plays an important role in intelligent traffic intersections. Target tracking technology can realize accurate identification and tracking of traffic participants at intersections (such as cars, pedestrians, bicycles, etc.) to conduct traffic optimization analysis such as traffic flow statistics, driving trajectory analysis, accident warning and intelligent signal control.

[0004] At present, target tracking is mainly carried out through point clouds obtained by radar. However, as the target moves in the intersection, the point clouds corresponding to some targets will be sparse, and targets with only a few point clouds will be difficult to identify, which may easily cause the target to be lost during the tracking process and reduce the durability of target tracking. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a target tracking method, device, equipment and storage medium based on multiple radars.

[0006] The present disclosure provides a multi-radar based target tracking method, the method comprising:

[0007] Determine target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area; wherein each current frame point cloud is obtained by scanning the target area by radar sensors with different scanning angles, and the number of current frame point clouds is the same as the number of radar sensors;

[0008] Based on the tracking object information corresponding to the previous frame point cloud, predict the initial tracking object information corresponding to each current frame point cloud;

[0009] Based on the initial tracking object information and the target detection object information, the initial tracking object and the detection object are matched to obtain a matching result;

[0010] Based on the matching results, the target tracking object and the target tracking object information corresponding to each current frame point cloud are determined, and target tracking is implemented based on the target tracking object and the target tracking object information.

[0011] In some embodiments, based on the matching results, determining the target tracking object and target tracking object information corresponding to each current frame point cloud includes:

[0012] If the matching result is that the initial tracking object does not match the detection object, then determine the number of points in each spliced ​​current frame point cloud contained in each initial grid area divided by the target area;

[0013] If the number of points corresponding to the target grid area where the initial tracking object is located is greater than the number threshold, the initial tracking object is determined as the target tracking object of the current frame, and based on the position information of the current frame point cloud in the target grid area and other information in the initial tracking object information except the position information, the target tracking object information is determined.

[0014] In some embodiments, based on the matching results, determining the target tracking object and target tracking object information corresponding to each current frame point cloud includes:

[0015] If the matching result is that the initial tracking object matches the detection object, the initial tracking object is determined as the target tracking object of the current frame, and the target detection object information is determined as the target tracking object information of the target tracking object.

[0016] In some embodiments, based on the matching results, determining the target tracking object and target tracking object information corresponding to each current frame point cloud includes:

[0017] If the matching result is that the detection object does not match the initial tracking object, then when the detection confidence in the target detection object information is greater than the first confidence, the detection object is determined as the target tracking object, and the target detection object information is determined as the target tracking object information of the target initial tracking object.

[0018] In some embodiments, after the matching result is that the detected object does not match the initial tracking object, the method further includes:

[0019] When the detection confidence in the target detection object information is less than or equal to the first confidence, the detection object and the target detection object information are discarded.

[0020] In some embodiments, before matching the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result, the method further includes:

[0021] If the detection confidence in the target detection object information is greater than the second confidence, retaining the detection object and the target detection object information; wherein the difference between the second confidence and the first confidence satisfies the difference threshold;

[0022] If the detection confidence in the target detection object information is less than or equal to the second confidence, the detection object and the target detection object information are discarded.

[0023] In some embodiments, after determining target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area, the method further includes:

[0024] If all current frame point clouds are first frame point clouds, the detection object in the target detection object information whose detection confidence is greater than the first confidence is determined as the target tracking object of the current frame, and the corresponding target detection object information is determined as the target tracking object information of the target tracking object.

[0025] In some embodiments, determining target detection object information of a detection object in the target area based on multiple current frame point clouds in the target area includes:

[0026] Input each current frame point cloud into the 3D object detection model respectively to generate a detection result consistent with the number of the current frame point cloud; wherein the detection result includes the initial detection object information of each detection object in the corresponding current frame point cloud;

[0027] The detection results are integrated to determine the target detection object information of each detection object contained in the target area.

[0028] In some embodiments, fusing the detection results to determine the target detection object information of each detection object contained in the target area includes:

[0029] Based on the posture relationship between each radar sensor, determine the posture conversion matrix;

[0030] Based on the posture transformation matrix, each detection result is transformed into the same coordinate system to generate the transformed detection result;

[0031] For any two converted detection results, matching is performed between the detection objects based on the initial detection object information, and the initial detection object information of each detection object that has not been successfully matched is determined as the target detection object information of the corresponding detection object, and based on the initial detection object information of the two detection objects that have been successfully matched, the retained detection object and the target detection object information of the retained detection object are determined.

[0032] In some embodiments, matching between the detection objects based on the initial detection object information includes:

[0033] Based on each initial detection object information in the two detection results, a distance matrix is ​​determined; wherein any element in the distance matrix represents the distance between any pair of detection objects in the two detection results;

[0034] The two detection objects corresponding to the elements in the distance matrix whose element values ​​are less than the distance threshold are determined as the two detection objects that have successfully matched.

[0035] In some embodiments, after determining that two detection objects corresponding to elements in the distance matrix whose element values ​​are less than the distance threshold are two detection objects that have successfully matched, the method further includes:

[0036] The Hungarian algorithm is used to perform a second match on each of the two successfully matched detection objects to update the two successfully matched detection objects.

[0037] In some embodiments, determining a distance matrix based on each initial detection object information in two detection results includes:

[0038] Based on the initial detection object information of any pair of detection objects in the two detection results, determining an intersection-over-union ratio of the corresponding two detection objects in a bird's-eye view as the distance;

[0039] Generate a distance matrix based on each intersection and union ratio.

[0040] In some embodiments, based on each initial detection object information of two detection objects that are successfully matched, determining the retained detection object and the target detection object information of the retained detection object includes:

[0041] From the two successfully matched detection objects, select a detection object with a larger detection confidence in the initial detection object information as the retained detection object;

[0042] The target detection object information of the retained detection object is determined based on the sum of the detection confidences in the initial detection object information of the two detection objects that are successfully matched.

[0043] In some embodiments, determining target detection object information of a detection object in the target area based on multiple current frame point clouds in the target area includes:

[0044] Splice the point clouds of each current frame to generate a spliced ​​point cloud corresponding to the target area;

[0045] The spliced ​​point cloud is input into the 3D target detection model to generate target detection object information of each detection object contained in the target area.

[0046] The present disclosure also provides a multi-radar based target tracking device, the device comprising:

[0047] A target detection object information determination module is used to determine the target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area; wherein each current frame point cloud is obtained by scanning the target area by radar sensors with different scanning angles, and the number of current frame point clouds is the same as the number of radar sensors;

[0048] An initial tracking object information prediction module, used to predict the initial tracking object information corresponding to each current frame point cloud based on the tracking object information corresponding to the previous frame point cloud;

[0049] An object matching module is used to match the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result;

[0050] The target tracking module is used to determine the target tracking object and target tracking object information corresponding to each current frame point cloud based on the matching result, and implement target tracking based on the target tracking object and target tracking object information.

[0051] In some embodiments, the target tracking module includes a first information determination submodule configured to:

[0052] If the matching result is that the initial tracking object does not match the detection object, then determine the number of points in each spliced ​​current frame point cloud contained in each initial grid area divided by the target area;

[0053] If the number of points corresponding to the target grid area where the initial tracking object is located is greater than the number threshold, the initial tracking object is determined as the target tracking object of the current frame, and based on the position information of the current frame point cloud in the target grid area and other information in the initial tracking object information except the position information, the target tracking object information is determined.

[0054] In some embodiments, the target tracking module includes a second information determination submodule for:

[0055] If the matching result is that the initial tracking object matches the detection object, the initial tracking object is determined as the target tracking object of the current frame, and the target detection object information is determined as the target tracking object information of the target tracking object.

[0056] In some embodiments, the target tracking module includes a third information determination submodule for:

[0057] If the matching result is that the detection object does not match the initial tracking object, then when the detection confidence in the target detection object information is greater than the first confidence, the detection object is determined as the target tracking object, and the target detection object information is determined as the target tracking object information of the target initial tracking object.

[0058] In some embodiments, the third information determination submodule is further used to:

[0059] After the matching result is that the detection object is not matched to the initial tracking object, when the detection confidence in the target detection object information is less than or equal to the first confidence, the detection object and the target detection object information are discarded.

[0060] In some embodiments, the device further includes a detection object filtering module, configured to:

[0061] Before matching the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result, if the detection confidence in the target detection object information is greater than the second confidence, retaining the detection object and the target detection object information; wherein the difference between the second confidence and the first confidence satisfies the difference threshold;

[0062] If the detection confidence in the target detection object information is less than or equal to the second confidence, the detection object and the target detection object information are discarded.

[0063] In some embodiments, the target tracking module is further configured to:

[0064] After determining the target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area, if each current frame point cloud is a first frame point cloud, the detection object in the target detection object information whose detection confidence is greater than the first confidence is determined as the target tracking object of the current frame, and the corresponding target detection object information is determined as the target tracking object information of the target tracking object.

[0065] In some embodiments, the target detection object information determination module includes:

[0066] A detection result generation submodule is used to input each current frame point cloud into a 3D object detection model to generate a detection result consistent with the number of the current frame point cloud; wherein the detection result includes initial detection object information of each detection object in the corresponding current frame point cloud;

[0067] The target detection object information determination submodule is used to fuse various detection results and determine the target detection object information of each detection object contained in the target area.

[0068] In some embodiments, the target detection object information determination submodule includes:

[0069] A posture conversion matrix determination unit, used for determining a posture conversion matrix based on the posture relationship between the radars;

[0070] A detection result conversion unit, used to convert each detection result to the same coordinate system based on the posture conversion matrix to generate a converted detection result;

[0071] The target detection object information determination unit is used to match the detection objects based on the initial detection object information for any two converted detection results, and determine the initial detection object information of each detection object that has not been successfully matched as the target detection object information of the corresponding detection object, and determine the retained detection object and the target detection object information of the retained detection object based on the initial detection object information of the two detection objects that have been successfully matched.

[0072] In some embodiments, the target detection object information determination unit is specifically used to:

[0073] Based on each initial detection object information in the two detection results, a distance matrix is ​​determined; wherein any element in the distance matrix represents the distance between any pair of detection objects in the two detection results;

[0074] The two detection objects corresponding to the elements in the distance matrix whose element values ​​are less than the distance threshold are determined as the two detection objects that have successfully matched.

[0075] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0076] After determining the two detection objects corresponding to the elements whose element values ​​in the distance matrix are less than the distance threshold as the two successfully matched detection objects, the Hungarian algorithm is used to perform secondary matching on the two successfully matched detection objects to update the two successfully matched detection objects.

[0077] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0078] Based on the initial detection object information of any pair of detection objects in the two detection results, determining an intersection-over-union ratio of the corresponding two detection objects in a bird's-eye view as the distance;

[0079] Generate a distance matrix based on each intersection and union ratio.

[0080] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0081] From the two successfully matched detection objects, select a detection object with a larger detection confidence in the initial detection object information as the retained detection object;

[0082] The target detection object information of the retained detection object is determined based on the sum of the detection confidences in the initial detection object information of the two detection objects that are successfully matched.

[0083] In some embodiments, the target detection object information determination module is specifically used to:

[0084] Splice the point clouds of each current frame to generate a spliced ​​point cloud corresponding to the target area;

[0085] The spliced ​​point cloud is input into the 3D target detection model to generate target detection object information of each detection object contained in the target area.

[0086] The present disclosure also provides an electronic device, the electronic device comprising:

[0087] Processor and memory;

[0088] The processor is used to execute the target tracking method described in any embodiment of the present disclosure by calling the program or instruction stored in the memory.

[0089] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a program or instruction, and the program or instruction enables a computer to execute the target tracking method described in any embodiment of the present disclosure.

[0090] The target tracking method, apparatus, device and storage medium provided by the embodiments of the present disclosure obtain target detection object information of each detection object contained therein from multiple current frame point clouds acquired by multiple radar sensors in the target area, and predict the initial tracking object information of the initial tracking object corresponding to each current frame point cloud from the tracking object information corresponding to the previous frame point cloud, and then match the detection object with the initial tracking object through the target detection object information and the initial tracking object information in the current frame to obtain a matching result; based on the matching result, determine the target tracking object and the target tracking object information corresponding to each current frame point cloud, and implement target tracking based on the target tracking object and the target tracking object information; realize enriching the point cloud of each object in the target area with the point cloud acquired by multiple radar sensors, increase the number of points of the tracking object in the edge area, thereby reducing the situation where the tracking object has only sparse point clouds, to a certain extent solve the problem of losing the tracking object due to sparse point clouds, and improve the persistence of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0092] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0093] Figure 1A schematic diagram of a flow chart of a target tracking method based on multiple radars provided in an embodiment of the present disclosure;

[0094] Figure 2 A schematic diagram of multiple radars and their scanning ranges in a traffic intersection provided by an embodiment of the present disclosure;

[0095] Figure 3 A flowchart of another multi-radar based target tracking method provided by an embodiment of the present disclosure;

[0096] Figure 4 A schematic diagram of a detailed process of fusing multiple detection results in a multi-radar based target tracking method provided in an embodiment of the present disclosure;

[0097] Figure 5 A schematic diagram of the structure of a multi-radar based target tracking device provided in an embodiment of the present disclosure;

[0098] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0099] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described in detail below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0100] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0101] The multi-radar based target tracking method provided by the embodiment of the present disclosure is mainly applicable to the scenario of using radar point cloud for target tracking, for example, it can be applicable to the scenario of dynamically tracking moving objects (such as pedestrians, bicycles, electric vehicles and / or vehicles, etc.) at traffic intersections through radar point cloud. The multi-radar based target tracking method provided by the embodiment of the present disclosure can be performed by a multi-radar based target tracking device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device with point cloud data processing capability. The electronic device can be, for example, a laptop computer, a desktop computer, a server, an edge computing platform, a server cluster, etc.

[0102] Figure 1 is a flow chart of a target tracking method based on multiple radars provided by an embodiment of the present disclosure. Figure 1 , the multi-radar based target tracking method specifically includes:

[0103] S110. Determine target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area.

[0104] Among them, the target area is the area to be processed for target tracking. The current frame point cloud is a frame of point cloud to be processed at the current moment. Each current frame point cloud is obtained by scanning the target area by radar sensors with different scanning angles, so the number of current frame point clouds is the same as the number of radar sensors. The detection object is an object obtained by target detection in the current frame point cloud, for example, it can be a pedestrian, bicycle, electric vehicle and vehicle in a traffic scene. The target detection object information is the final characterization information of the detection object. In the embodiment of the present disclosure, the target detection object information can be represented as p = [x, y, z, l, w, h, theta, score], where [x, y, z] represents the center point position of the detection object, [l, w, h] represents the size of the detection object, theta represents the orientation angle of the detection object, and score represents the confidence that the detection object exists in the target area (which can be simply referred to as detection confidence).

[0105] Specifically, see Figure 2 , a radar sensor can be deployed at each of the four road intersections in the traffic intersection. The examples in the figure are primary radar, secondary radar 1, secondary radar 2 and secondary radar 3. Each radar sensor can scan within a certain range according to its scanning angle of view to obtain the current frame point cloud corresponding to the scanning angle of view. Therefore, by setting up multiple radar sensors with different scanning angles in the target area (such as a traffic intersection), the same number of current frame point clouds can be collected through these radar sensors. The electronic device can receive each current frame point cloud directly from the radar sensor. Alternatively, each radar sensor stores each current frame point cloud obtained by its scan in a storage space (local or cloud). The electronic device obtains each current frame point cloud by accessing the storage space.

[0106] Then, the electronic device performs target detection processing based on each current frame point cloud as basic data, and obtains each non-repetitive detection object and its target detection object information in the target area contained in each current frame point cloud.

[0107] In some embodiments, the process of performing target detection processing on multiple current frame point clouds can be: inputting each current frame point cloud into a 3D target detection model respectively to generate a detection result consistent with the number of current frame point clouds; fusing each detection result to determine the target detection object information of each detection object contained in the target area.

[0108] Specifically, the electronic device can input each current frame point cloud into a pre-trained 3D target detection model. After the model is operated, the detection result corresponding to each current frame point cloud can be output. The detection result includes the detection object detected from the corresponding current frame point cloud and its detection object information (i.e., the initial detection object information). Then, the electronic device can fuse the various detection results according to the posture relationship between the radar sensors to obtain the non-repetitive detection objects in the target area and the target detection object information of each detection object. In this way, the point cloud can be batched for target detection, which can reduce the computational complexity of the 3D target detection model to a certain extent and improve the efficiency of obtaining the detection object and its target detection object information.

[0109] In some embodiments, the above-mentioned 3D target detection model can be the feature extraction network and the detection head network in the CenterPoint model, while the speed prediction part and the center point position improvement part of the network are discarded to ensure the accuracy of target detection, reduce the model calculation amount and improve the target detection efficiency.

[0110] In other embodiments, the process of performing target detection processing on multiple current frame point clouds can be: splicing the current frame point clouds to generate a spliced ​​point cloud corresponding to the target area; inputting the spliced ​​point cloud into the 3D target detection model to generate target detection object information for each detection object contained in the target area.

[0111] Specifically, the electronic device can first splice the point clouds of each current frame in three-dimensional space to obtain a spliced ​​point cloud of the target area (i.e., a spliced ​​point cloud). Then, the spliced ​​point cloud is input into the 3D target detection model, and after model calculation, each non-repeated detection object in the target area and the target detection object information of each detection object can be output.

[0112] S120 . Predicting initial tracking object information of initial tracking objects corresponding to point clouds of each current frame based on the tracking object information corresponding to the point cloud of the previous frame.

[0113] The previous frame point cloud is the point cloud of the frame immediately before the current frame, which is the spliced ​​point cloud of the target area. The tracking object information is the information of the tracking object determined in the previous frame point cloud, which may include the center point position, size, orientation angle and confidence of the tracking object as a tracking object (referred to as tracking confidence). The initial tracking object is the tracking object preliminarily determined from the current frame point cloud. The initial tracking object information is the information of the preliminarily determined initial tracking object.

[0114] Specifically, the electronic device uses the tracking object information corresponding to the previous frame point cloud and the Kalman algorithm to predict the information of the tracking object in the current frame point cloud, and obtains the non-repetitive initial tracking object and its initial tracking object information contained in each current frame point cloud. The initial tracking object information may include the center point position, size, orientation angle and tracking confidence of the initial tracking object.

[0115] S130: Based on the initial tracking object information and the target detection object information, the initial tracking object and the detection object are matched to obtain a matching result.

[0116] Specifically, the electronic device uses a matching algorithm in related technology (such as a matching algorithm for calculating the intersection-and-union ratio) based on the initial tracking object information and the target detection object information to pair each detection object in the current frame point cloud with each initial tracking object, and obtains matching results: detection objects and initial tracking objects that match each other, initial tracking objects that are not matched to detection objects, and detection objects that are not matched to initial tracking objects. For example, the overlap between each detection object and each initial tracking object can be calculated based on the above initial tracking object information and target detection object information. Combined with a pre-set overlap threshold, the above matching result can be obtained. The overlap threshold here can be set empirically based on whether the two polygonal objects are one object.

[0117] S140 . Determine the target tracking object and the target tracking object information corresponding to each current frame point cloud based on the matching result, and implement target tracking based on the target tracking object and the target tracking object information.

[0118] The target tracking object is the tracking object finally determined in each current frame point cloud. The target tracking object information is the representation information of the target tracking object, which may include the center point position, size, orientation angle and tracking confidence of the target tracking object.

[0119] Specifically, the electronic device can remove, retain, and correct the target detection object information of each detection object and its target detection object information according to the matching results obtained above, and obtain the target tracking objects and their target tracking object information that are non-repetitive in the target area corresponding to each current frame point cloud. Then, the electronic device uses the target tracking objects and their target tracking object information of the current frame as basic data and performs target tracking processing for the next frame according to the above process.

[0120] The target tracking method based on multiple radars provided in the above embodiments of the present disclosure is as follows:

[0121] The target detection object information of each detection object contained therein is obtained by using multiple current frame point clouds collected by multiple radar sensors in the target area, and the initial tracking object information of the initial tracking object corresponding to each current frame point cloud is predicted by the tracking object information corresponding to the previous frame point cloud, and then the detection object and the initial tracking object are matched by the target detection object information and the initial tracking object information in the current frame to obtain a matching result; based on the matching result, the target tracking object and the target tracking object information corresponding to each current frame point cloud are determined, and target tracking is implemented based on the target tracking object and the target tracking object information; the point clouds collected by multiple radar sensors are used to enrich the point clouds of each object in the target area, and the number of points of the tracking object in the edge area is increased, thereby reducing the situation where the tracking object has only sparse point clouds, which solves the problem of losing the tracking object due to sparse point clouds to a certain extent and improves the persistence of target tracking.

[0122] In some embodiments, based on the above technical solution, if each current frame point cloud is a first frame point cloud, then the process of determining the target tracking object and its information can be: determining the detection object in the target detection object information whose detection confidence is greater than the first confidence as the target tracking object corresponding to the current frame point cloud, and determining the corresponding target detection object information as the target tracking object information of the target tracking object.

[0123] The first confidence level is a preset confidence level threshold, which is used to determine whether an object can be used as a target tracking object. Therefore, in order to ensure the accuracy of the target tracking object, the first confidence level can be set to a larger confidence level. For example, the first confidence level can be set to a value of 0.6 or above.

[0124] Specifically, for the first frame point cloud, it is impossible to predict the tracking object. However, the business requirement in the traffic scene is to track objects appearing on the road, which has great randomness and uncertainty. Therefore, the tracking object cannot be specified in advance. In this case, at least one detection object with a higher confidence among the detection objects obtained can be determined as the target tracking object. The electronic device can compare the detection confidence and the first confidence in each target detection object information. The detection object whose detection confidence is greater than the first confidence and its target detection object information are determined as the target tracking object and its target tracking object information.

[0125] In other embodiments, if the business requirement is to track a specific object, the tracking object can be pre-specified. In this case, the detection object and its target detection object information that successfully matches the specified tracking object among the detection objects corresponding to the first frame point cloud can be determined as the target tracking object and its target tracking object information.

[0126] Figure 3: is a flowchart of another multi-radar target tracking method provided by an embodiment of the present disclosure. Based on the above embodiments, the multi-radar target tracking method further refines "determining the target tracking object and target tracking object information corresponding to each current frame point cloud based on the matching results". On the basis of the above, the processing process of filtering the detection object can be further added. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. See Figure 3 , the multi-radar based target tracking method includes:

[0127] S301. Determine target detection object information of a detection object in a target area based on multiple current frame point clouds in the target area.

[0128] S302 : Predicting initial tracking object information of initial tracking objects corresponding to each current frame point cloud based on the tracking object information corresponding to the previous frame point cloud.

[0129] S303: Determine whether the detection confidence in the target detection object information is greater than a second confidence.

[0130] The second confidence level is another preset confidence level threshold, which is used to preliminarily filter the detection objects with low confidence levels obtained in the current frame point cloud. Considering that the detection confidence level fluctuates as the object moves in the traffic scene. For example, Figure 2 When vehicle 1 enters the middle of the intersection, it is in the middle of the scanning range of multiple radar sensors, so more point clouds can be obtained, so its detection confidence is higher; and when vehicle 1 is about to drive to Figure 2 Before the position shown, it is only at the edge of the scanning range of the secondary radar 1, and only a small number of points corresponding to it can be obtained, then the detection confidence of vehicle 1 will be low. In order to adapt to the actual fluctuation of detection confidence, the second confidence can be set to a smaller value, so as to retain the detection objects as much as possible on the basis of filtering out extremely unreliable detection objects, which can avoid the problem of eliminating too many detection objects and causing more initial tracking objects to fail to match the detection objects, resulting in the omission of some tracking objects, and avoid the problem of low detection confidence of the retained detection objects and low accuracy of the tracking object information updated by them, thereby improving the stability and accuracy of object tracking. For example, the first confidence can be set to 0.1 or 0.2, etc.

[0131] Exemplarily, the difference between the second confidence and the first confidence satisfies a difference threshold. The difference threshold may be a preset confidence difference. In combination with the functional description of the first confidence and the second confidence, the difference between the two may be maximized to balance the stability of the retained detection object and the accuracy of the tracked object.

[0132] Specifically, the electronic device may use the second confidence level to perform preliminary filtering on each detection object before the matching operation. That is, the detection confidence level in each target detection object information is compared with the second confidence level, and the processing method for the detection object is determined according to the comparison result. If the detection confidence level is greater than the second confidence level, S304 is executed; if the detection confidence level is less than or equal to the second confidence level, S305 is executed.

[0133] S304: retain the detection object and target detection object information.

[0134] Specifically, when the detection confidence is greater than the second confidence, it indicates that the corresponding detection object is relatively reliable, and the detection object and its target detection object information are retained.

[0135] S305: discard the detection object and target detection object information.

[0136] Specifically, when the detection confidence is less than or equal to the second confidence, it indicates that the reliability of the detection object is low and it is not of reference value, so the detection object and its target detection object information are eliminated.

[0137] S306: Based on the initial tracking object information and the target detection object information, the initial tracking object and the detection object are matched to obtain a matching result.

[0138] Specifically, if the matching result is that the initial tracking object does not match the detection object, execute S307; if the matching result is that the initial tracking object matches the detection object, execute S310; if the matching result is that the detection object does not match the initial tracking object, execute S311.

[0139] S307: Determine the number of points in each spliced ​​current frame point cloud contained in each initial grid area divided into the target area, and determine whether the number of points corresponding to the target grid area where the initial tracking object is located is greater than a number threshold.

[0140] The initial grid area is a local area obtained by gridding the target area. The quantity threshold is the number of pre-set points, which can be empirically set according to the number of points that are required to represent an object.

[0141] Specifically, on the basis that multiple current frame point clouds enrich the number of points of objects in the target area, there may still be a situation where only a sparse point cloud is obtained for a certain object. A sparse point cloud will make it difficult to detect the object. Therefore, in the embodiment of the present disclosure, the target area can be divided into multiple initial grid areas of smaller areas, and the number of points in each spliced ​​current frame point cloud in each initial grid area can be counted as a basis for subsequent judgment of whether a certain undetected object actually exists.

[0142] In some embodiments, the process of rasterizing the target area and determining the number of points may specifically be: according to the preset rasterization range [x_min, x_max, y_min, y_max, z_min, z_max] and the size of each grid [v_x, v_y, v_z], the target area is divided into initial grid areas. Then, the globally unique grid id corresponding to each point in the current frame point cloud is calculated to establish a corresponding relationship between each point in the current frame point cloud and the grid id, so as to locate the specific initial grid area according to the position of the object in the subsequent process.

[0143] For example, the 3D grid index ID of point p (three-dimensional coordinates represented by [p_x, p_y, p_z]) in the point cloud of the current frame can be calculated by the following formula (1).

[0144]

[0145] Then, taking the x-axis coordinate as the guide, the 3D grid index ID of each point is converted into a globally unique grid ID according to formula (2).

[0146]

[0147] After that, for the current frame point cloud, the number of points falling into each initial grid area is counted. In this way, the corresponding relationship between each point in the current frame point cloud, the globally unique grid id and the number of points can be established for query use in subsequent processes.

[0148] It should be noted that the above method for determining the grid ID is only an example, and its implementation method may also be other methods that can determine the global grid ID of the grid where a point is located. For example, in formula (2), the grid ID may be calculated based on the y-axis coordinate or the z-axis coordinate.

[0149] In specific implementation, if the above matching results determine that a certain initial tracking object does not match the detection object, then the initial tracking object may really disappear in the target area, or it may appear in an area with sparse point clouds in the target area and not be detected. Figure 2 The vehicle 1 in the image is currently located behind the primary radar, and its point cloud is all from the point cloud of the secondary radar 1 on the opposite side. Due to the long distance, its point cloud is relatively sparse, resulting in the 3D target detection model failing to detect the vehicle 1. If the initial tracking object is directly discarded, the problem of tracking loss will occur. Therefore, in the embodiment of the present disclosure, the number of points in each initial grid area determined in the above process can be used to determine whether the initial tracking object still exists in the target area.

[0150] The electronic device can query the correspondence between the point coordinates constructed above, the globally unique grid ID and the number of points according to the center point position in the initial tracking object information, and obtain the number of points in the initial grid area (i.e., the target grid area) where the initial tracking object is located. Then, compare whether the number of points is greater than the number threshold. If so, execute S308; if not, execute S309.

[0151] In some embodiments, the electronic device may stitch each current frame point cloud in real time and count the number of points in each initial grid area.

[0152] In other embodiments, the electronic device may perform splicing of each current frame point cloud and count the number of points in each grid area before this step. In this step, the number of points in each initial grid area may be determined by querying the obtained information.

[0153] S308: Determine the initial tracking object as the target tracking object of the current frame, and determine the target tracking object information of the target tracking object based on the position information of the current frame point cloud in the target grid area and other information in the initial tracking object information except the position information.

[0154] Specifically, if the number of queried points is greater than the number threshold, it means that the initial tracking object still exists in the target grid area, and the initial tracking object is determined as the target tracking object. In addition, the center point position of each point in the current frame point cloud existing in the target grid area can be used to update the center point position in the corresponding initial tracking object information, while retaining other information in the corresponding initial tracking object information, such as size, orientation angle and tracking confidence, to obtain the target tracking object information, so as to ensure the tracking accuracy and persistence of the initial tracking object.

[0155] S309: discard the initial tracking object and the initial tracking object information.

[0156] Specifically, if the number of queried points is less than or equal to the number threshold, it means that there is indeed no object in the target grid area, and the initial tracking object and its information are discarded.

[0157] S310: Determine the initial tracking object as the target tracking object corresponding to the current frame point cloud, and determine the target detection object information as the target tracking object information of the target tracking object.

[0158] Specifically, when the initial tracking object successfully matches a certain detection object, it means that the initial tracking object has been successfully tracked from the current frame point cloud. In this case, since the reliability of the target detection object information is higher than the predicted initial tracking object information, the successfully matched detection object can be determined as the target tracking object, and the target detection object information of the successfully matched detection object is used to update the initial tracking object information of the successfully matched initial tracking object as its target tracking object information.

[0159] S311. Determine whether the detection confidence in the target detection object information is greater than a first confidence.

[0160] Specifically, when the detected object does not match any of the initial tracking objects, it means that the detected object is not a previously tracked object, but it may be a newly appeared tracking object. For example, a new moving object enters the traffic intersection. In this case, the detected object cannot be directly discarded, but the reliability of the detected object is judged based on the comparison result of its detection confidence and the first confidence, and then the processing method of the detected object is determined. If the comparison result is that the detection confidence is greater than the first confidence, S312 is executed; otherwise, S313 is executed.

[0161] S312: Determine the detected object as a target tracking object, and determine the target detection object information as target tracking object information of the target initial tracking object.

[0162] Specifically, if the detection confidence of the unpaired detection object is greater than the first confidence, it means that the detection object has a high reliability, and it is very likely that a new tracking object has appeared. Therefore, the electronic device adds the detection object as a target tracking object in the current frame, and determines its target detection object information as the target tracking object information of the newly added target tracking object.

[0163] S313: discard the detection object and target detection object information.

[0164] Specifically, if the detection confidence of the unpaired detection object is less than or equal to the first confidence, it means that the reliability of the detection object is not high. Then, in order to ensure the accuracy of the target tracking object, it is not used as the target tracking object in the current frame, but the detection object and its target detection object information are removed.

[0165] It should be noted that the execution order between S302 and S303-S306 is not limited here. S303-S306 may be executed first and then S302, or S302 and S303-S306 may be executed in parallel. In addition, S307-S309, S310 and S311-S313 may be executed one by one or two by two. When at least two of the parts are executed, the order of serial execution between the parts is not limited.

[0166] The above technical solution of the embodiment of the present disclosure determines that when the initial tracking object fails to match the detection object, it may be determined that the predicted initial tracking object disappears in the current frame or the point cloud of the initial tracking object is relatively sparse and has not been detected. Therefore, the number of points of the point cloud of the initial tracking object in the target grid area divided by the target area can be obtained, and when the number of points exceeds the number threshold, it is determined that the initial tracking object still exists in the current frame but has not been detected, and it is determined as the target tracking object, and the position information of the point cloud in the target grid area and other information in the initial tracking object information are used as the target tracking object information; this realizes that when the tracking object has only a sparse point cloud, It can still be tracked effectively, further avoiding the problem of losing the tracking object in the case of sparse point cloud, thereby further improving the persistence of target tracking; using the successfully matched detection object and its target detection object information to update the target tracking object information of the successfully matched target tracking object, thereby improving the accuracy of the target tracking object; by using different first confidence levels and second confidence levels in the tracking process, the adaptability to the detection confidence levels with large fluctuations is improved, the stability of the target tracking object information update during the tracking process is improved, and the credibility of the newly added target tracking objects is improved, thereby further improving the comprehensiveness, stability and accuracy of the target tracking object.

[0167] Figure 4 1 is a detailed flow chart of fusing multiple detection results in a multi-radar based target tracking method provided by an embodiment of the present disclosure. The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here. Figure 4 The process of fusing multiple detection results in the multi-radar based target tracking method specifically includes:

[0168] S410: Determine a posture conversion matrix based on the posture relationship between the radar sensors.

[0169] Specifically, for any two radar sensors, the posture relationship between the two can be used to determine the rotation matrix R and the translation vector t shown in the following formula (3), and thus obtain the posture transformation matrix between the two radar sensors:

[0170]

[0171] Among them, R represents a 3*3 rotation matrix, and t represents a 3*1 translation vector. R is determined by the Euler angles (α, β, γ) between the coordinate systems of the two radar sensors, t is determined by the positions of the two radar sensors, and dx, dy, and dz represent the distances on the same coordinate axes in the coordinate systems of the two radar sensors.

[0172] S420: Based on the posture transformation matrix, each detection result is transformed into the same coordinate system to generate a transformed detection result.

[0173] Specifically, using the above-mentioned posture conversion matrix, the detection result in the coordinate system of one radar sensor can be converted to the coordinate system of another radar sensor through the following formula (4):

[0174]

[0175] According to the above process, the detection results of the coordinate systems of the radar sensors can be transferred to the same coordinate system to obtain the converted detection results.

[0176] S430. For any two converted detection results, matching is performed between the detection objects based on the initial detection object information, and the initial detection object information of each detection object that has not been successfully matched is determined as the target detection object information of the corresponding detection object, and based on the initial detection object information of the two detection objects that have been successfully matched, the retained detection object and the target detection object information of the retained detection object are determined.

[0177] Specifically, there are duplications between the detection objects in each converted detection result. Therefore, deduplication processing can be performed on each detection object in any two converted detection results. During deduplication processing, each initial detection object information in the two detection results is traversed to match each detection object. If any two detection objects are successfully matched, one of the detection objects and its initial detection object information is retained as the corresponding target detection object information. If there are detection objects that have not been successfully matched, it means that there are no duplications in these detection objects, then these unmatched detection objects are all retained, and the corresponding initial detection object information is determined as the corresponding target detection object information.

[0178] In some embodiments, "matching each detection object based on each initial detection object information" in S430 includes: determining a distance matrix based on each initial detection object information in two detection results; and determining two detection objects corresponding to elements in the distance matrix whose element values ​​are less than a distance threshold as two detection objects that have successfully matched.

[0179] Among them, any element in the distance matrix represents the distance between any pair of detected objects in two detection results. The distance threshold is a pre-set distance, which is used to determine whether two objects are duplicate objects.

[0180] Specifically, in this embodiment, the distance between any two detection objects is calculated for matching. Then the distance between any two detection objects in the two transformed detection results can form a distance matrix M as shown below:

[0181]

[0182] Among them, d ij Represents the i-th detection object p in a detection result i and the jth detection object p in another detection result j The distance between.

[0183] Compare each element value in the distance matrix M with the distance threshold. If an element value is less than the distance threshold, it means that the two detection objects corresponding to the element value are duplicate objects and the two are matched successfully. On the contrary, if the element value is greater than or equal to the distance threshold, it means that the two detection objects corresponding to the element value are far apart and belong to two independent detection objects.

[0184] By executing the above process for any two converted detection results, the pairing processing of the detection objects in each converted detection result can be completed, and multiple groups of paired detection objects and multiple unpaired detection objects can be obtained.

[0185] In some embodiments, the distance between the two objects can be characterized by the degree of overlap between the two objects. Then, the process of determining the distance matrix can be: based on the initial detection object information of any pair of detection objects in the two detection results, determine the intersection and union ratio of the corresponding two detection objects in the top view as the distance; generate the distance matrix based on each intersection and union ratio.

[0186] Specifically, in this embodiment, the intersection-over-union (IOU) ratio between the two detection objects to be matched can be calculated to characterize the degree of overlap between the two detection objects. The larger the IOU ratio, the greater the degree of overlap between the two detection objects and the smaller the distance between them. In this way, the element value of the above distance matrix is ​​d ij =1-IOU(p i ,p j ).

[0187] In some embodiments, there is a situation where one detection object among the above-mentioned multiple groups of paired detection objects is successfully paired with multiple detection objects at the same time, that is, a detection object is simultaneously a detection object with multiple detection objects that are close to the surrounding area. In this way, there is a possibility of incorrect matching. Therefore, secondary matching can be performed on these paired detection objects to obtain a better matching pair. At this time, the electronic device can use the Hungarian algorithm to perform secondary matching on the two detection objects that have successfully matched, so that each detection object is paired with another better detection object, and the above matching pairs are updated with the results of the secondary matching, eliminating the above-mentioned one-to-many situation, and improving the fusion accuracy of multiple converted detection results.

[0188] In some embodiments, "determining the retained detection object and the target detection object information of the retained detection object based on the initial detection object information of the two successfully matched detection objects" in S430 specifically includes: from the two successfully matched detection objects, screening a detection object with a larger detection confidence in the initial detection object information as the retained detection object; determining the target detection object information of the retained detection object based on the sum of the detection confidences in the initial detection object information of the two successfully matched detection objects.

[0189] Specifically, for paired detection objects, the detection object with a higher detection confidence can be retained to further improve the accuracy of the detection object. Considering that if two radar sensors simultaneously identify the same detection object, regardless of whether its detection confidence is high or not, it means that the detection object is likely to exist in the target area. Therefore, the detection confidence of the retained detection object can be improved by adding the two detection confidences. In this way, the detection confidence new_score of the retained detection object is as shown in formula (5):

[0190] new_score=min(1,score_1+score_2)(5)

[0191] Among them, score_1 and score_2 respectively represent the detection confidence of the two paired detection objects.

[0192] The center point position, size and orientation angle in the target detection object information of the retained detection object adopt the corresponding values ​​in the initial detection object information of the retained detection object.

[0193] Figure 5 A schematic diagram of the structure of a multi-radar based target tracking device provided in an embodiment of the present disclosure.

[0194] like Figure 5 As shown, the multi-radar based target tracking device 500 includes:

[0195] The target detection object information determination module 510 is used to determine the target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area; wherein each current frame point cloud is obtained by scanning the target area with radar sensors of different scanning angles, and the number of current frame point clouds is the same as the number of radar sensors;

[0196] An initial tracking object information prediction module 520, configured to predict initial tracking object information of an initial tracking object corresponding to each current frame point cloud based on the tracking object information corresponding to the previous frame point cloud;

[0197] The object matching module 530 is used to match the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result;

[0198] The target tracking module 540 is used to determine the target tracking object and the target tracking object information corresponding to each current frame point cloud based on the matching result, and implement target tracking based on the target tracking object and the target tracking object information.

[0199] A target tracking device based on multiple radars provided by an embodiment of the present disclosure can obtain target detection object information of each detection object contained therein from multiple current frame point clouds collected by multiple radar sensors in the target area, and predict initial tracking object information of the initial tracking object corresponding to each current frame point cloud from the tracking object information corresponding to the previous frame point cloud, and then match the detection object and the initial tracking object with the target detection object information and the initial tracking object information in the current frame to obtain a matching result; based on the matching result, determine the target tracking object and the target tracking object information corresponding to each current frame point cloud, and implement target tracking based on the target tracking object and the target tracking object information; realize enriching the point cloud of each object in the target area with the point cloud collected by multiple radar sensors, increase the number of points of the tracking object in the edge area, thereby reducing the situation where the tracking object has only sparse point clouds, to a certain extent solve the problem of losing the tracking object due to sparse point clouds, and improve the persistence of target tracking.

[0200] In some embodiments, the target tracking module 540 includes a first information determination submodule for:

[0201] If the matching result is that the initial tracking object does not match the detection object, then determine the number of points in each spliced ​​current frame point cloud contained in each initial grid area divided by the target area;

[0202] If the number of points corresponding to the target grid area where the initial tracking object is located is greater than the number threshold, the initial tracking object is determined as the target tracking object of the current frame, and based on the position information of the current frame point cloud in the target grid area and other information in the initial tracking object information except the position information, the target tracking object information is determined.

[0203] In some other embodiments, the target tracking module 540 includes a second information determination submodule, which is used to:

[0204] If the matching result is that the initial tracking object matches the detection object, the initial tracking object is determined as the target tracking object of the current frame, and the target detection object information is determined as the target tracking object information of the target tracking object.

[0205] In some further embodiments, the target tracking module 540 includes a third information determination submodule, which is used to:

[0206] If the matching result is that the detection object does not match the initial tracking object, then when the detection confidence in the target detection object information is greater than the first confidence, the detection object is determined as the target tracking object, and the target detection object information is determined as the target tracking object information of the target initial tracking object.

[0207] In some embodiments, the third information determination submodule is further used to:

[0208] After the matching result is that the detection object is not matched to the initial tracking object, when the detection confidence in the target detection object information is less than or equal to the first confidence, the detection object and the target detection object information are discarded.

[0209] In some embodiments, the device further includes a detection object filtering module, configured to:

[0210] Before matching the initial tracking object and the detection object based on the initial tracking object information and the target detection object information to obtain a matching result, if the detection confidence in the target detection object information is greater than the second confidence, retaining the detection object and the target detection object information; wherein the difference between the second confidence and the first confidence satisfies the difference threshold;

[0211] If the detection confidence in the target detection object information is less than or equal to the second confidence, the detection object and the target detection object information are discarded.

[0212] In some embodiments, the target tracking module 540 is further configured to:

[0213] After determining the target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area, if each current frame point cloud is a first frame point cloud, the detection object in the target detection object information whose detection confidence is greater than the first confidence is determined as the target tracking object of the current frame, and the corresponding target detection object information is determined as the target tracking object information of the target tracking object.

[0214] In some embodiments, the target detection object information determination module 510 includes:

[0215] A detection result generation submodule is used to input each current frame point cloud into a 3D object detection model to generate a detection result consistent with the number of the current frame point cloud; wherein the detection result includes initial detection object information of each detection object in the corresponding current frame point cloud;

[0216] The target detection object information determination submodule is used to fuse various detection results and determine the target detection object information of each detection object contained in the target area.

[0217] In some embodiments, the target detection object information determination submodule includes:

[0218] A posture conversion matrix determination unit, used for determining a posture conversion matrix based on the posture relationship between each radar sensor;

[0219] A detection result conversion unit, used to convert each detection result to the same coordinate system based on the posture conversion matrix to generate a converted detection result;

[0220] The target detection object information determination unit is used to match the detection objects based on the initial detection object information for any two converted detection results, and determine the initial detection object information of each detection object that has not been successfully matched as the target detection object information of the corresponding detection object, and determine the retained detection object and the target detection object information of the retained detection object based on the initial detection object information of the two detection objects that have been successfully matched.

[0221] In some embodiments, the target detection object information determination unit is specifically used to:

[0222] Based on each initial detection object information in the two detection results, a distance matrix is ​​determined; wherein any element in the distance matrix represents the distance between any pair of detection objects in the two detection results;

[0223] The two detection objects corresponding to the elements in the distance matrix whose element values ​​are less than the distance threshold are determined as the two detection objects that have successfully matched.

[0224] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0225] After determining the two detection objects corresponding to the elements whose element values ​​in the distance matrix are less than the distance threshold as the two successfully matched detection objects, the Hungarian algorithm is used to perform secondary matching on the two successfully matched detection objects to update the two successfully matched detection objects.

[0226] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0227] Based on the initial detection object information of any pair of detection objects in the two detection results, determining an intersection-over-union ratio of the corresponding two detection objects in a bird's-eye view as the distance;

[0228] Generate a distance matrix based on each intersection and union ratio.

[0229] In some embodiments, the target detection object information determination unit is further specifically configured to:

[0230] From the two successfully matched detection objects, select a detection object with a larger detection confidence in the initial detection object information as the retained detection object;

[0231] The target detection object information of the retained detection object is determined based on the sum of the detection confidences in the initial detection object information of the two detection objects that are successfully matched.

[0232] In some embodiments, the target detection object information determination module 510 is specifically used to:

[0233] Splice the point clouds of each current frame to generate a spliced ​​point cloud corresponding to the target area;

[0234] The spliced ​​point cloud is input into the 3D target detection model to generate target detection object information of each detection object contained in the target area.

[0235] The multi-radar based target tracking device provided in the embodiments of the present disclosure can execute the multi-radar based target tracking method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0236] It is worth noting that in the above-mentioned embodiment of the multi-radar based target tracking device, the various modules, sub-modules and units included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional modules / sub-modules / units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure.

[0237] Figure 6 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and a memory 602 .

[0238] The processor 601 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0239] The memory 602 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 601 may run the program instructions to implement the multi-radar-based target tracking method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as current frame point cloud, target detection object information, and target tracking object information may also be stored in the computer-readable storage medium.

[0240] In one example, the electronic device 600 may further include: an input device 603 and an output device 604, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 603 may include, for example, a keyboard, a mouse, etc. The output device 604 may output various information to the outside, including each frame of point cloud and its corresponding target tracking object information, etc. The output device 604 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0241] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device 600 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 600 may also include any other appropriate components.

[0242] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the multi-radar based target tracking method provided by the embodiment of the present disclosure.

[0243] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0244] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the multi-radar based target tracking method provided by the embodiment of the present disclosure.

[0245] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0246] It should be noted that the terms used in the present disclosure are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present disclosure specification and claims, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The term "and / or" includes any and all combinations of one or more related listed items. Relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method or device including the elements.

[0247] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on multiple radars, characterized in that: include: Determine target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area; wherein each of the current frame point clouds is obtained by scanning the target area with radar sensors of different scanning angles, and the number of the current frame point clouds is the same as the number of the radar sensors; Predicting initial tracking object information of initial tracking objects corresponding to each of the current frame point clouds based on the tracking object information corresponding to the previous frame point cloud; If the detection confidence in the target detection object information is greater than the second confidence, retaining the detection object and the target detection object information; wherein the difference between the second confidence and the first confidence satisfies the difference threshold; If the detection confidence in the target detection object information is less than or equal to the second confidence, discarding the detection object and the target detection object information; Based on the initial tracking object information and the target detection object information, matching the initial tracking object and the detection object to obtain a matching result; Based on the matching result, determining the target tracking object and the target tracking object information corresponding to each of the current frame point clouds, and performing target tracking based on the target tracking object and the target tracking object information; Wherein, determining the target tracking object and target tracking object information corresponding to each current frame point cloud based on the matching result includes: If the matching result is that the detection object does not match the initial tracking object, then when the detection confidence in the target detection object information is greater than a first confidence, the detection object is determined as a target tracking object, and the target detection object information is determined as the target tracking object information of the target initial tracking object; Wherein, determining the target detection object information of the detection object in the target area based on multiple current frame point clouds in the target area includes: Input each of the current frame point clouds into a 3D object detection model to generate a detection result consistent with the number of the current frame point clouds; wherein the detection result includes initial detection object information of each detection object in the corresponding current frame point cloud; Fusing the detection results to determine the target detection object information of each detection object contained in the target area; Wherein, fusing the detection results to determine the target detection object information of each detection object contained in the target area includes: Determining a posture conversion matrix based on the posture relationship between the radar sensors; Based on the posture transformation matrix, each of the detection results is transformed into the same coordinate system to generate a transformed detection result; For any two of the converted detection results, matching is performed between the detection objects based on the initial detection object information, and the initial detection object information of each detection object that has not been successfully matched is determined as the target detection object information of the corresponding detection object, and based on the initial detection object information of the two detection objects that have been successfully matched, the retained detection object and the target detection object information of the retained detection object are determined; Wherein, determining the retained detection object and the target detection object information of the retained detection object based on the initial detection object information of the two detection objects that are successfully matched includes: From the two detection objects that are successfully matched, select the detection object with a larger detection confidence in the initial detection object information as the retained detection object; The target detection object information of the retained detection object is determined based on the sum of the detection confidences in the initial detection object information of the two detection objects that are successfully matched.

2. The method according to claim 1, characterized in that The determining, based on the matching result, the target tracking object and the target tracking object information corresponding to each of the current frame point clouds includes: If the matching result is that the initial tracking object does not match the detection object, determining the number of points in each of the spliced ​​current frame point clouds contained in each of the initial grid areas divided by the target area; If the number of points corresponding to the target grid area where the initial tracking object is located is greater than the number threshold, the initial tracking object is determined as the target tracking object of the current frame, and the target tracking object information of the target tracking object is determined based on the position information of the current frame point cloud in the target grid area and other information in the initial tracking object information except the position information.

3. The method according to claim 1, characterized in that The determining, based on the matching result, the target tracking object and the target tracking object information corresponding to each of the current frame point clouds includes: If the matching result is that the initial tracking object matches the detection object, the initial tracking object is determined as the target tracking object of the current frame, and the target detection object information is determined as the target tracking object information of the target tracking object.

4. The method according to claim 1, characterized in that: After the matching result is that the detected object is not matched to the initial tracking object, the method further includes: When the detection confidence in the target detection object information is less than or equal to the first confidence, the detection object and the target detection object information are discarded.

5. The method according to claim 1, characterized in that After determining the target detection object information of the detection object in the target area based on the multiple current frame point clouds in the target area, the method further includes: If all the current frame point clouds are first frame point clouds, the detection object in the target detection object information whose detection confidence is greater than the first confidence is determined as the target tracking object of the current frame, and the corresponding target detection object information is determined as the target tracking object information of the target tracking object.

6. The method according to claim 1, characterized in that The matching between the detection objects based on the initial detection object information includes: Based on each of the initial detection object information in the two detection results, a distance matrix is ​​determined; wherein any element in the distance matrix represents the distance between any pair of the detection objects in the two detection results; The two detection objects corresponding to the elements in the distance matrix whose element values ​​are smaller than the distance threshold are determined as the two detection objects that have successfully matched.

7. The method according to claim 6, characterized in that After determining the two detection objects corresponding to the elements in the distance matrix whose element values ​​are less than the distance threshold as the two detection objects that have successfully matched, the method further includes: The Hungarian algorithm is used to perform a second match on the two successfully matched detection objects to update the two successfully matched detection objects.

8. The method according to claim 6, characterized in that The determining of the distance matrix based on each of the initial detection object information in the two detection results comprises: Based on the initial detection object information of any pair of the detection objects in the two detection results, determining an intersection-over-union ratio of the corresponding two detection objects in a top-down perspective as the distance; The distance matrix is ​​generated based on each of the intersection-over-union ratios.

9. The method according to claim 1, characterized in that: The step of determining target detection object information of a detection object in the target area based on a plurality of current frame point clouds in the target area includes: Splicing the point clouds of the current frames to generate a spliced ​​point cloud corresponding to the target area; The spliced ​​point cloud is input into a 3D target detection model to generate target detection object information of each detection object contained in the target area.

10. A target tracking device based on multiple radars, characterized in that: include: A target detection object information determination module, configured to determine the target detection object information of the detection object in the target area based on a plurality of current frame point clouds in the target area; wherein each of the current frame point clouds is obtained by scanning the target area with a radar sensor of a different scanning angle, and the number of the current frame point clouds is the same as the number of the radar sensors; An initial tracking object information prediction module, used to predict the initial tracking object information of the initial tracking object corresponding to each of the current frame point clouds based on the tracking object information corresponding to the previous frame point cloud; A detection object filtering module, configured to retain the detection object and the target detection object information if the detection confidence in the target detection object information is greater than a second confidence; wherein the difference between the second confidence and the first confidence satisfies a difference threshold; and discard the detection object and the target detection object information if the detection confidence in the target detection object information is less than or equal to the second confidence; An object matching module, configured to match the initial tracking object with the detection object based on the initial tracking object information and the target detection object information to obtain a matching result; A target tracking module, used to determine the target tracking object and the target tracking object information corresponding to each of the current frame point clouds based on the matching result, and implement target tracking based on the target tracking object and the target tracking object information; Wherein, the target tracking module includes a third information determination submodule, which is used to: If the matching result is that the detection object does not match the initial tracking object, then when the detection confidence in the target detection object information is greater than a first confidence, the detection object is determined as a target tracking object, and the target detection object information is determined as the target tracking object information of the target initial tracking object; Among them, the target detection object information determination module includes: A detection result generation submodule, used to input each of the current frame point clouds into a 3D object detection model, and generate a detection result consistent with the number of the current frame point clouds; wherein the detection result includes initial detection object information of each detection object in the corresponding current frame point cloud; A target detection object information determination submodule, used to fuse the detection results and determine the target detection object information of each detection object contained in the target area; Among them, the target detection object information determination submodule includes: A posture conversion matrix determination unit, used for determining a posture conversion matrix based on the posture relationship between the radar sensors; A detection result conversion unit, used to convert each of the detection results into the same coordinate system based on the posture conversion matrix to generate a converted detection result; a target detection object information determination unit, configured to match the detection objects based on the initial detection object information for any two of the converted detection results, and determine the initial detection object information of each of the detection objects that failed to match as the target detection object information of the corresponding detection object, and determine the retained detection object and the target detection object information of the retained detection object based on the initial detection object information of the two detection objects that successfully matched; The target detection object information determination unit is specifically used for: From the two detection objects that are successfully matched, select the detection object with a larger detection confidence in the initial detection object information as the retained detection object; The target detection object information of the retained detection object is determined based on the sum of the detection confidences in the initial detection object information of the two detection objects that are successfully matched.

Images