Target association fusion method, apparatus, vehicle and storage medium

Abstract

This application relates to a target association fusion method, apparatus, vehicle, and storage medium, comprising: caching clustered detection targets and deep learning detection targets to a first detection target cache queue and a second detection target cache queue, respectively; acquiring timestamp-matched clustered detection targets and deep learning detection targets, and placing them into a preset vector container; if the preset vector container is a first preset value, using the deep learning detection target or the clustered detection target as the associated fused target; if the preset vector container is a second preset value, calculating a matching matrix; if the matching matrix is ​​empty, using the deep learning detection target as the associated fused target; if the matching matrix is ​​not empty, traversing the matching matrix and using the deep learning detection target and clustered detection targets that meet preset screening conditions as the associated fused target, thereby outputting a more stable and accurate detection target for unified tracking processing.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a target association fusion method, device, vehicle, and storage medium. Background Technology

[0002] In the field of LiDAR target detection and tracking technology, target detection typically follows two technical approaches: traditional clustering detection methods and deep learning detection methods. A common practice is to track targets detected by clustering and those detected by deep learning separately, then output the identified targets for each, and finally perform targeted tracking processing based on the characteristics of the clustered and deep learning-detected targets.

[0003] However, the drawbacks are also quite obvious. First, there is no distinction between clustering and deep learning detection. In general, deep learning detection results are more accurate and the output target is more stable. Clustering, due to the limitations of the method, has a larger fluctuation in the output target. Therefore, the conventional approach is to use deep learning as the main method and clustering as a redundant auxiliary method. Second, tracking and processing the targets of clustering and deep learning detection separately increases the processing complexity and resource consumption. Summary of the Invention

[0004] This application provides a target association fusion method, apparatus, vehicle, and storage medium, which solves the problems of traditional LiDAR target detection and tracking, which do not perform association fusion on single-frame clustering and deep learning-detected targets before tracking, and cannot perform single-shot fusion tracking on clustered and deep learning-detected targets. It can fully utilize the stability and accuracy of deep learning-detected targets, while using the advantages of low false negative rate and false recognition rate of clustering detection to make up for the shortcomings of deep learning detection, thereby outputting more stable and accurate detected targets, providing good input for subsequent single-shot target tracking.

[0005] The first aspect of this application provides a target association fusion method, comprising the following steps: obtaining clustered detection targets and deep learning detection targets, and caching the clustered detection targets in a first detection target cache queue and caching the deep learning detection targets in a second detection target cache queue; obtaining timestamp-matched clustered detection targets and deep learning detection targets from the first and second detection target cache queues, and placing the timestamp-matched clustered detection targets and deep learning detection targets into a preset vector container; if the preset vector container is a first preset value, then using the deep learning detection targets or the clustered detection targets as the associated fusion targets; if the preset vector container is a second preset value, then calculating a matching matrix based on the distance between all clustered detection targets and all deep learning detection targets in the preset vector container, and when the matching matrix is ​​empty, using the deep learning detection targets as the associated fusion targets; when the matching matrix is ​​not empty, traversing the matching matrix and using the deep learning detection targets and the clustered detection targets that meet preset screening conditions as the associated fusion targets.

[0006] Based on the above technical means, the problems of traditional LiDAR target detection and tracking, such as not performing correlation and fusion of single-frame clustering and deep learning-detected targets before tracking, and not being able to perform single-time fusion tracking of clustered and deep learning-detected targets, are solved. It can make full use of the stability and accuracy of deep learning-detected targets, while using the advantages of low false negative rate and false recognition rate of clustering detection to make up for the shortcomings of deep learning detection, thereby outputting more stable and accurate detected targets.

[0007] Further, the step of calculating the matching matrix based on the distances between all clustered detection targets and all deep learning detection targets in the preset vector container includes: calculating the distances between all clustered detection targets and all deep learning detection targets in the preset vector container to obtain a Euclidean distance matrix and an IoU distance matrix; calculating a weighted distance matrix based on the Euclidean distance matrix and the IoU distance matrix; and using a preset Hungarian algorithm to calculate the weighted distance matrix to obtain the matching matrix.

[0008] Based on the above technical means, the matching matrix is ​​obtained by calculating the distance between all clustered detection targets and all deep learning detection targets, which improves the accuracy of multi-target tracking.

[0009] Further, the step of calculating the weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix includes: calculating the weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix according to a preset weighted distance matrix, wherein the preset weighted distance matrix is:

[0010] C = a * M_dist + b * M_iou;

[0011] Where a and b are weights, M_dist is the Euclidean distance matrix, and M_iou is the Iou distance matrix.

[0012] Based on the aforementioned technical means, a pre-set weighted distance matrix is ​​calculated to output a more accurate target detection result.

[0013] Further, the step of using the deep learning detection target and the clustering detection target that meet the preset screening conditions as the target after association and fusion includes: traversing the matching matrix, obtaining the element with a value of 1 in the matching matrix, and deleting the target in the clustering detection target that corresponds to the column number where the element is located; and using the remaining detection targets of the deep learning detection target and the clustering detection target as the target after association and fusion.

[0014] Based on the above technical means, by deleting clustered classification targets and merging identical targets, a fused single-frame detection target is obtained for unified tracking processing.

[0015] Further, the step of placing the timestamp-matched clustering detection target and deep learning detection target into a preset vector container includes: obtaining the tail msg_ai of the deep learning detection target, and traversing from the tail of the first detection target cache queue to obtain msg_seg; determining whether the timestamp of msg_ai is equal to the timestamp of msg_seg; if the timestamp of msg_ai is equal to the timestamp of msg_seg, then first placing msg_ai into the preset vector container, and then placing msg_seg into the preset vector container; otherwise, determining whether the timestamp of msg_ai is less than the timestamp of msg_seg; if the timestamp of msg_ai is less than the timestamp of msg_seg... If the timestamp is the same as the timestamp of msg_ai, then it is determined whether there is a msg_seg in the first detection target cache queue with the same timestamp as msg_ai. If there is a msg_seg with the same timestamp as msg_ai, both msg_ai and the msg_seg with the same timestamp as msg_ai are put into the preset vector container. If there is no msg_seg with the same timestamp as msg_ai, msg_ai is put into the preset vector container. Otherwise, it is determined whether the timestamp of msg_ai is greater than the timestamp of msg_seg. If the timestamp of msg_ai is greater than the timestamp of msg_seg, then msg_ai is directly put into the preset vector container.

[0016] Based on the above technical means, the detection targets of the unified frame original point cloud are obtained by judging the timestamp of the clustering and deep learning output targets.

[0017] Further, caching the clustering output detection targets to a first detection target cache queue and caching the deep learning output detection targets to a second detection target cache queue includes: determining the first queue length of the first detection target cache queue and the second queue length of the second detection target cache queue; when the first queue length is greater than a preset length threshold, deleting the head element of the first queue length, and when the second queue length is greater than the preset length threshold, deleting the head element of the second queue length, so that both the first queue length and the second queue length are the preset length threshold.

[0018] Based on the above technical means, by making both the length of the first queue and the length of the second queue a preset length threshold, it is convenient to obtain the clustering and deep learning detection targets generated from the same frame of original point cloud data.

[0019] A second aspect of this application provides a target association fusion apparatus, comprising: a caching module, configured to acquire detection targets output by clustering and detection targets output by deep learning, and cache the detection targets output by clustering to a first detection target caching queue, and cache the detection targets output by deep learning to a second detection target caching queue; an input module, configured to acquire time-stamp-matched clustered detection targets and deep learning detection targets from the first detection target caching queue and the second detection target caching queue, and place the time-stamp-matched clustered detection targets and deep learning detection targets into a preset vector container; and a calculation module. If the preset vector container is a first preset value, then the deep learning detection target or the clustering detection target is used as the associated fused target; if the preset vector container is a second preset value, then a matching matrix is ​​calculated based on the distance between all clustering detection targets and all deep learning detection targets in the preset vector container, and when the matching matrix is ​​empty, the deep learning detection target is used as the associated fused target; when the matching matrix is ​​not empty, the matching matrix is ​​traversed, and the deep learning detection target and the clustering detection target that meets the preset screening conditions are used as the associated fused target.

[0020] Furthermore, the calculation module is also used to: calculate the distance between all clustered detection targets and all deep learning detection targets in the preset vector container to obtain a Euclidean distance matrix and an IOU distance matrix; calculate a weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix, and use a preset Hungarian algorithm to calculate the weighted distance matrix to obtain the matching matrix.

[0021] Furthermore, the calculation module is also configured to: calculate the weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix, including: calculating the weighted distance matrix based on a preset weighted distance matrix, wherein the preset weighted distance matrix is:

[0022] C = a * M_dist + b * M_iou;

[0023] Where a and b are weights, M_dist is the Euclidean distance matrix, and M_iou is the Iou distance matrix.

[0024] Furthermore, the calculation module is also used to: traverse the matching matrix, obtain the elements with a value of 1 in the matching matrix, and delete the target in the cluster detection target that corresponds to the column number where the element is located; and use the remaining detection targets of the deep learning detection target and the cluster detection target as the target after association fusion.

[0025] Furthermore, the input module is also configured to: obtain the tail msg_ai of the deep learning detection target, and traverse from the tail of the first detection target cache queue to obtain msg_seg; determine whether the timestamp of msg_ai is equal to the timestamp of msg_seg; if the timestamp of msg_ai is equal to the timestamp of msg_seg, then first put msg_ai into the preset vector container, and then put msg_seg into the preset vector container; otherwise, determine whether the timestamp of msg_ai is less than the timestamp of msg_seg; if the timestamp of msg_ai is less than the timestamp of msg_seg, then determine whether the first detection target cache queue... Check if a `msg_seg` with the same timestamp as `msg_ai` exists in the target cache queue. If a `msg_seg` with the same timestamp as `msg_ai` exists, put both `msg_ai` and the `msg_seg` with the same timestamp as `msg_ai` into the preset vector container. If no `msg_seg` with the same timestamp as `msg_ai` exists, put `msg_ai` into the preset vector container. Otherwise, determine if the timestamp of `msg_ai` is greater than the timestamp of `msg_seg`. If the timestamp of `msg_ai` is greater than the timestamp of `msg_seg`, directly put `msg_ai` into the preset vector container.

[0026] Furthermore, the caching module is also configured to: determine the first queue length of the first detection target cache queue and the second queue length of the second detection target cache queue; when the first queue length is greater than a preset length threshold, delete the head element of the first queue length, and when the second queue length is greater than the preset length threshold, delete the head element of the second queue length, so that the first queue length and the second queue length are both the preset length threshold.

[0027] A third aspect of this application provides a vehicle comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target association fusion method as described in the above embodiments.

[0028] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the target association fusion method as described in the above embodiments.

[0029] Therefore, this application caches the clustering output and the deep learning output of the detected targets in a first and a second detection target cache queue, respectively. It also obtains the timestamp-matched clustering and deep learning detection targets and places them into a preset vector container. When the preset vector container has a certain value, either the deep learning detection target or the clustering detection target is used as the fused target. Alternatively, a matching matrix is ​​calculated based on the distance between all clustering detection targets and all deep learning detection targets. If the matching matrix is ​​empty, the deep learning detection target is used as the fused target; otherwise, the deep learning detection target and the clustering detection target that meets the preset screening conditions are used as the fused target. This solves the problems of traditional LiDAR target tracking, which fails to fuse single-frame clustering and deep learning detection targets before tracking, making single-shot fusion tracking impossible. It fully utilizes the stability and accuracy of deep learning target detection while leveraging the low false negative and false positive rates of clustering detection to compensate for the shortcomings of deep learning detection, thus outputting more stable and accurate detection targets and providing good input for subsequent single-shot target tracking.

[0030] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0031] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0032] Figure 1This is a flowchart of a target association fusion method provided according to an embodiment of this application;

[0033] Figure 2 This is a flowchart of a target association fusion method according to an embodiment of this application;

[0034] Figure 3 This is a block diagram of a target association fusion apparatus according to an embodiment of this application;

[0035] Figure 4 This is a structural schematic diagram of a vehicle according to an embodiment of this application.

[0036] Explanation of reference numerals in the attached drawings: 10-Target correlation fusion device, 100-Cache module, 200-Input module, 300-Computing module, 401-Memory, 402-Processor, 403-Communication interface. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0038] The target association fusion method, apparatus, vehicle, and storage medium of this application are described below with reference to the accompanying drawings. Addressing the problem mentioned in the background art that traditional LiDAR target tracking does not perform association fusion on clustered and deep learning-detected targets in a single frame before tracking, thus failing to perform single-shot fusion tracking on clustered and deep learning-detected targets, this application provides a target association fusion method. In this method, the clustered output targets and the deep learning output targets are cached in a first target cache queue and a second target cache queue, respectively. Timestamp-matched clustered and deep learning-detected targets are obtained and placed into a preset vector container. When the preset vector container has a certain value, either the deep learning-detected target or the clustered-detected target is used as the fused target. Alternatively, a matching matrix is ​​calculated based on the distance between all clustered-detected targets and all deep learning-detected targets. If the matching matrix is ​​empty, the deep learning-detected target is used as the fused target; otherwise, the deep learning-detected target and clustered-detected targets that meet preset screening conditions are used as the fused target. This solves the problem that traditional LiDAR target detection and tracking does not perform correlation and fusion of single-frame clustering and deep learning-detected targets before tracking, and cannot perform single-shot fusion tracking of clustered and deep learning-detected targets. It fully utilizes the stability and accuracy of deep learning-detected targets, while using the advantages of low false negative and false recognition rates of clustering detection to compensate for the shortcomings of deep learning detection, thereby outputting more stable and accurate detected targets, providing good input for subsequent single-shot target tracking.

[0039] Specifically, Figure 1 This is a flowchart of a target association fusion method provided in an embodiment of this application.

[0040] like Figure 1 As shown, the target association fusion method includes the following steps:

[0041] In step S101, the detection targets output by clustering and the detection targets output by deep learning are obtained, and the detection targets output by clustering are cached in the first detection target cache queue, and the detection targets output by deep learning are cached in the second detection target cache queue.

[0042] Since the frequencies of upstream clustering and deep learning in generating detection targets may differ, in order to obtain the clustering detection targets and deep learning detection targets generated from the same frame of original point cloud data, it is necessary to cache the clustering output detection targets and deep learning output detection targets for a period of time.

[0043] In this embodiment, two independent queues, queue_msg_seg (i.e., the first detection target cache queue) and queue_msg_ai (the second detection target cache queue), are used to cache the detection targets output by clustering and the detection targets output by deep learning, respectively.

[0044] Furthermore, in some embodiments, caching the clustering output detection targets to a first detection target cache queue and caching the deep learning output detection targets to a second detection target cache queue includes: determining the first queue length of the first detection target cache queue and the second queue length of the second detection target cache queue; deleting the head element of the first queue when the first queue length is greater than a preset length threshold, and deleting the head element of the second queue when the second queue length is greater than the preset length threshold, so that both the first queue length and the obtained second queue length are the preset length threshold.

[0045] The preset length threshold can be a threshold set by the user, a threshold obtained through a limited number of experiments, or a threshold obtained through a limited number of computer simulations. For ease of understanding, the preset length threshold is set to 10 frames.

[0046] Specifically, such as Figure 2 As shown, step S1 involves generating the target detection cache queue. Whenever a new target is entered into the first and second target detection cache queues, the queue lengths of the first and second target detection cache queues are determined. If the length of the first target detection cache queue is greater than 10 frames, the first element of the first queue is deleted. Similarly, if the length of the second target detection cache queue is greater than 10 frames, the first element of the second queue is deleted, i.e., the earliest detected target message is deleted, ensuring that both the first and second queue lengths are 10 frames.

[0047] In step S102, timestamp-matched clustered detection targets and deep learning detection targets are obtained from the first detection target cache queue and the second detection target cache queue, and the timestamp-matched clustered detection targets and deep learning detection targets are placed into a preset vector container.

[0048] If the time difference between the target output by clustering detection and the target output by deep learning detection is too large, they cannot be correlated and merged. Therefore, in order to obtain the detected targets of the same frame of original point cloud, it is necessary to first obtain the clustering and deep learning detected targets with matching timestamps from the first and second detection target cache queues.

[0049] Further, in some embodiments, placing the timestamp-matched clustering detection target and the deep learning detection target into a preset vector container includes: obtaining the tail of the deep learning detection target's queue, msg_ai, and traversing from the tail of the first detection target cache queue to obtain msg_seg; determining whether the timestamp of msg_ai is equal to the timestamp of msg_seg; if the timestamp of msg_ai is equal to the timestamp of msg_seg, then first placing msg_ai into the preset vector container, and then placing msg_seg into the preset vector container; otherwise, determining whether the timestamp of msg_ai is less than the timestamp of msg_seg; if the timestamp of msg_ai is less than the timestamp of msg_seg, then... If the timestamp of _seg is used, it is determined whether there is a msg_seg with the same timestamp as msg_ai in the first detection target cache queue. If there is a msg_seg with the same timestamp as msg_ai, both msg_ai and the msg_seg with the same timestamp as msg_ai are put into a preset vector container. If there is no msg_seg with the same timestamp as msg_ai, msg_ai is put into the preset vector container. Otherwise, it is determined whether the timestamp of msg_ai is greater than the timestamp of msg_seg. If the timestamp of msg_ai is greater than the timestamp of msg_seg, msg_ai is directly put into the preset vector container.

[0050] Specifically, such as Figure 2 As shown, step S2 is to align the detection targets in the clustering output with the detection targets in the deep learning output in time. First, iterate from the tail of the second detection target cache queue to obtain the tail element msg_ai. Then, iterate from the tail of the first detection target cache queue to obtain the element msg_seg. If the timestamp of msg_ai is equal to the timestamp of msg_seg, push msg_ai_msg into the preset vector container vec_msg, and then push msg_seg into the preset vector container vec_msg. If the timestamp of msg_ai is less than the timestamp of msg_seg, search the first detection target cache queue for a msg_seg with the same timestamp as msg_ai. If a msg_seg with the same timestamp as msg_ai exists in the first detection target cache queue, put msg_ai and the msg_seg with the same timestamp as msg_ai into the preset vector container vec_msg in turn. If the timestamp of msg_ai is greater than the timestamp of msg_seg, put msg_ai directly into the preset vector container vec_msg.

[0051] In step S103, if the preset vector container is a first preset value, then the deep learning detection target or the clustering detection target is used as the target after association and fusion; if the preset vector container is a second preset value, then the matching matrix is ​​calculated based on the distance between all clustering detection targets and all deep learning detection targets in the preset vector container, and when the matching matrix is ​​empty, the deep learning detection target is used as the target after association and fusion; when the matching matrix is ​​not empty, the matching matrix is ​​traversed, and the deep learning detection target and the clustering detection target that meets the preset screening conditions are used as the target after association and fusion.

[0052] The first preset value and the second preset value can be thresholds set by the user, thresholds obtained through a limited number of experiments, or thresholds obtained through a limited number of computer simulations; no specific limitations are made here.

[0053] It is understandable that the target after association and fusion is calculated based on the size of the preset vector container vec_msg. If the size of the preset vector container vec_msg is the first preset value, then the deep learning detection target msg_ai or the clustering detection target msg_seg is used as the target after association and fusion.

[0054] Furthermore, in some embodiments, calculating the matching matrix based on the distances between all clustered detection targets and all deep learning detection targets in a preset vector container includes: calculating the distances between all clustered detection targets and all deep learning detection targets in a preset vector container to obtain a Euclidean distance matrix and an IoU distance matrix; calculating a weighted distance matrix based on the Euclidean distance matrix and the IoU distance matrix; and using a preset Hungarian algorithm to calculate the weighted distance matrix to obtain the matching matrix.

[0055] It should be understood that, such as Figure 2 As shown, step S3 is to generate the cost matrix. If the size of the preset vector container vec_msg is the second preset value, then the distance between all clustered detection targets and all deep learning detection targets in the preset vector container is calculated to obtain the Euclidean distance matrix M_dist and the IoU distance matrix M_iou. The cost matrix has a size of M×N, where M is the number of targets output by deep learning detection and N is the position of the detection targets output by clustering.

[0056] Further, in some embodiments, calculating the weighted distance matrix based on the Euclidean distance matrix and the IoU distance matrix includes: calculating the weighted distance matrix based on the Euclidean distance matrix and the IoU distance matrix according to a preset weighted distance matrix, wherein the preset weighted distance matrix is:

[0057] C = a * M_dist + b * M_iou;

[0058] Where a and b are weights, M_dist is the Euclidean distance matrix, and M_iou is the Iou distance matrix.

[0059] Specifically, such as Figure 2 As shown, step S4 is the matching matrix calculation. The obtained Euclidean distance matrix M_dist and IOU distance matrix M_iou are input into the preset weighted distance matrix, and the matching matrix M of the preset weighted distance matrix C is calculated using the preset Hungarian algorithm.

[0060] If the matching matrix M is empty, the deep learning detection target msg_ai is directly output as the target after association and fusion.

[0061] Furthermore, in some embodiments, the deep learning detection target and the clustering detection target that meet the preset screening conditions are used as the associated and fused targets, including: traversing the matching matrix, obtaining the elements with a value of 1 in the matching matrix, and deleting the targets in the clustering detection targets that correspond to the column number of the element; and using the remaining detection targets of the deep learning detection target and the clustering detection target as the associated and fused targets.

[0062] The preset filtering condition is the target that is detected after deleting the target whose column number corresponds to the element with a value of 1 in the matching matrix.

[0063] Specifically, such as Figure 2 As shown, step S5 is target association and merging. When the matching matrix M is not empty, the matching matrix M is traversed to obtain the elements with a value of 1 in the matching matrix M (i.e., M(i,j)=1). Then, the target j corresponding to the column number of the clustering detection target msg_seg with a value of 1 is deleted. Finally, the remaining detection targets in the deep learning detection target msg_ai and the clustering detection target msg_seg are used as the targets after association and fusion.

[0064] According to the target association fusion method proposed in this application, the clustering output and the deep learning output are cached in a first target cache queue and a second target cache queue, respectively. Timestamp-matched clustering and deep learning targets are obtained and placed into a preset vector container. When the preset vector container has a certain value, either the deep learning target or the clustering target is used as the fused target. Alternatively, a matching matrix is ​​calculated based on the distance between all clustering targets and all deep learning targets. If the matching matrix is ​​empty, the deep learning target is used as the fused target; otherwise, the deep learning target and the clustering target that meets the preset screening conditions are used as the fused target. This solves the problems of traditional LiDAR target tracking, which fails to perform single-frame clustering and deep learning target association fusion before tracking, making single-shot fusion tracking impossible. It fully utilizes the stability and accuracy of deep learning target detection, while leveraging the low false negative and false positive rates of clustering detection to compensate for the shortcomings of deep learning detection, thus outputting a more stable and accurate target, providing a good input for subsequent single-shot target tracking.

[0065] Next, the target association fusion apparatus proposed according to the embodiments of this application is described with reference to the accompanying drawings.

[0066] Figure 3 This is a block diagram of the target association fusion device according to an embodiment of this application.

[0067] like Figure 3 As shown, the target association fusion device 10 includes: a cache module 100, an input module 200, and a calculation module 300.

[0068] The system includes a caching module 100, which acquires the detection targets output by clustering and the detection targets output by deep learning, and caches the clustering output detection targets in a first detection target cache queue and the deep learning output detection targets in a second detection target cache queue. An input module 200 acquires the timestamp-matched clustering detection targets and deep learning detection targets from the first and second detection target cache queues, and places them into a preset vector container. A calculation module 300, if the preset vector container is a first preset value, uses either the deep learning detection target or the clustering detection target as the associated and fused target; if the preset vector container is a second preset value, it calculates a matching matrix based on the distances between all clustering detection targets and all deep learning detection targets in the preset vector container, and when the matching matrix is ​​empty, uses the deep learning detection target as the associated and fused target; when the matching matrix is ​​not empty, it iterates through the matching matrix and uses the deep learning detection target and the clustering detection targets that meet preset screening conditions as the associated and fused targets.

[0069] Furthermore, in some embodiments, the calculation module 300 is also used to: calculate the distance between all clustered detection targets and all deep learning detection targets in a preset vector container to obtain a Euclidean distance matrix and an IOU distance matrix; calculate a weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix, and use a preset Hungarian algorithm to calculate the weighted distance matrix to obtain a matching matrix.

[0070] Furthermore, in some embodiments, the calculation module 300 is also configured to: calculate a weighted distance matrix based on the Euclidean distance matrix and the IoU distance matrix, including: calculating the weighted distance matrix based on a preset weighted distance matrix, wherein the preset weighted distance matrix is:

[0071] C = a * M_dist + b * M_iou;

[0072] Where a and b are weights, M_dist is the Euclidean distance matrix, and M_iou is the Iou distance matrix.

[0073] Furthermore, in some embodiments, the calculation module 300 is also used to: traverse the matching matrix, obtain the elements with a value of 1 in the matching matrix, and delete the targets in the clustering detection targets that correspond to the column number where the element is located; and use the remaining detection targets of the deep learning detection targets and the clustering detection targets as the targets after association and fusion.

[0074] Further, in some embodiments, the input module 200 is also configured to: obtain the tail of the deep learning detection target msg_ai, and traverse from the tail of the first detection target cache queue to obtain msg_seg; determine whether the timestamp of msg_ai is equal to the timestamp of msg_seg; if the timestamp of msg_ai is equal to the timestamp of msg_seg, then first put msg_ai into a preset vector container, and then put msg_seg into the preset vector container; otherwise, determine whether the timestamp of msg_ai is less than the timestamp of msg_seg; if the timestamp of msg_ai is less than the timestamp of msg_seg, then determine The first step is to check if a `msg_seg` with the same timestamp as `msg_ai` exists in the target cache queue. If a `msg_seg` with the same timestamp as `msg_ai` exists, both `msg_ai` and the `msg_seg` with the same timestamp as `msg_ai` are placed into a preset vector container. If no `msg_seg` with the same timestamp as `msg_ai` exists, `msg_ai` is placed into the preset vector container. Otherwise, it is determined whether the timestamp of `msg_ai` is greater than the timestamp of `msg_seg`. If the timestamp of `msg_ai` is greater than the timestamp of `msg_seg`, `msg_ai` is directly placed into the preset vector container.

[0075] Furthermore, in some embodiments, the caching module 100 is also used to: determine the first queue length of the first detection target cache queue and the second queue length of the second detection target cache queue; when the first queue length is greater than a preset length threshold, delete the head element in the first queue length, and when the second queue length is greater than the preset length threshold, delete the head element in the second queue length, so that both the first queue length and the obtained second queue length are the preset length threshold.

[0076] It should be noted that the foregoing explanation of the target association fusion method embodiment also applies to the target association fusion device of this embodiment, and will not be repeated here.

[0077] According to the target association fusion device proposed in this application, the clustered detection targets and the deep learning detection targets are cached in a first detection target cache queue and a second detection target cache queue, respectively. Timestamp-matched clustered detection targets and deep learning detection targets are obtained and placed into a preset vector container. When the preset vector container has a certain value, either the deep learning detection target or the clustered detection target is used as the fused target. Alternatively, a matching matrix is ​​calculated based on the distance between all clustered detection targets and all deep learning detection targets. If the matching matrix is ​​empty, the deep learning detection target is used as the fused target; otherwise, the deep learning detection target and the clustered detection target that meets the preset screening conditions are used as the fused target. This solves the problems of traditional LiDAR target tracking, which fails to perform association fusion on single-frame clustered and deep learning detection targets before tracking, making it impossible to perform single-pass fusion tracking on only clustered and deep learning detection targets. It fully utilizes the stability and accuracy of deep learning detection targets, while leveraging the low false negative and false positive rates of clustered detection to compensate for the shortcomings of deep learning detection, thereby outputting more stable and accurate detection targets and providing good input for subsequent single-pass target tracking.

[0078] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:

[0079] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0080] When the processor 402 executes the program, it implements the target association fusion method provided in the above embodiments.

[0081] Furthermore, the vehicle also includes:

[0082] Communication interface 403 is used for communication between memory 401 and processor 402.

[0083] The memory 401 is used to store computer programs that can run on the processor 402.

[0084] The memory 401 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0085] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0087] Processor 402 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0088] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target association fusion method described above.

[0089] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0090] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0091] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0092] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0093] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0094] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A target association fusion method, characterized in that, Includes the following steps: Obtain the detection targets output by clustering and the detection targets output by deep learning, and cache the detection targets output by clustering to a first detection target cache queue, and cache the detection targets output by deep learning to a second detection target cache queue; The clustered detection targets and deep learning detection targets with matching timestamps are obtained from the first detection target cache queue and the second detection target cache queue, and the clustered detection targets and deep learning detection targets with matching timestamps are put into a preset vector container; as well as If the preset vector container is a first preset value, then the deep learning detection target or the clustering detection target is used as the associated and fused target; if the preset vector container is a second preset value, then a matching matrix is ​​calculated based on the distance between all clustering detection targets and all deep learning detection targets in the preset vector container, and when the matching matrix is ​​empty, the deep learning detection target is used as the associated and fused target; when the matching matrix is ​​not empty, the matching matrix is ​​traversed, and the deep learning detection target and the clustering detection target that meets the preset screening conditions are used as the associated and fused target. The step of calculating the matching matrix based on the distances between all clustered detection targets and all deep learning detection targets in the preset vector container includes: Calculate the distances between all clustered detection targets and all deep learning detection targets in the preset vector container to obtain the Euclidean distance matrix and the IOU distance matrix; The weighted distance matrix is ​​calculated based on the Euclidean distance matrix and the IOU distance matrix, and the matching matrix is ​​obtained by calculating the weighted distance matrix using a preset Hungarian algorithm.

2. The method according to claim 1, characterized in that, The step of calculating the weighted distance matrix based on the Euclidean distance matrix and the IOU distance matrix includes: Based on a preset weighted distance matrix, a weighted distance matrix is ​​calculated according to the Euclidean distance matrix and the IOU distance matrix, wherein the preset weighted distance matrix is: C = a * M_dist + b * M_iou; Where a and b are weights, M_dist is the Euclidean distance matrix, and M_iou is the Iou distance matrix.

3. The method according to claim 1, characterized in that, The step of using the deep learning detection target and the clustering detection target that meets the preset screening conditions as the associated fused target includes: Traverse the matching matrix, obtain the elements with a value of 1 in the matching matrix, and delete the targets in the cluster detection targets that correspond to the column number of the element; The remaining detection targets of the deep learning detection target and the clustering detection target are used as the target after association and fusion.

4. The method according to claim 1, characterized in that, The step of placing the timestamp-matched clustering detection target and the deep learning detection target into a preset vector container includes: Obtain the tail msg_ai of the deep learning detection target, and traverse from the tail of the first detection target cache queue to obtain msg_seg; Determine whether the timestamp of msg_ai is equal to the timestamp of msg_seg; If the timestamp of msg_ai is equal to the timestamp of msg_seg, then first put msg_ai into the preset vector container, and then put msg_seg into the preset vector container; otherwise, determine whether the timestamp of msg_ai is less than the timestamp of msg_seg. If the timestamp of msg_ai is less than the timestamp of msg_seg, then it is determined whether there is a msg_seg with the same timestamp as msg_ai in the first detection target cache queue. If there is a msg_seg with the same timestamp as msg_ai, msg_ai and the msg_seg with the same timestamp as msg_ai are put into the preset vector container. If there is no msg_seg with the same timestamp as msg_ai, msg_ai is put into the preset vector container. Otherwise, it is determined whether the timestamp of msg_ai is greater than the timestamp of msg_seg. If the timestamp of msg_ai is greater than the timestamp of msg_seg, then msg_ai is directly put into the preset vector container.

5. The method according to claim 1, characterized in that, Cache the clustering output detection targets to a first detection target cache queue, and cache the deep learning output detection targets to a second detection target cache queue, including: Determine the first queue length of the first detection target cache queue and the second queue length of the second detection target cache queue; When the length of the first queue is greater than the preset length threshold, the head element of the first queue is deleted, and when the length of the second queue is greater than the preset length threshold, the head element of the second queue is deleted, so that the length of the first queue and the length of the second queue are both the preset length threshold.

6. A target association and fusion device, characterized in that, include: A caching module is used to obtain the detection targets output by clustering and the detection targets output by deep learning, and cache the detection targets output by clustering to a first detection target cache queue, and cache the detection targets output by deep learning to a second detection target cache queue; The input module is used to obtain the timestamp-matched clustered detection targets and deep learning detection targets from the first detection target cache queue and the second detection target cache queue, and put the timestamp-matched clustered detection targets and deep learning detection targets into a preset vector container; as well as The calculation module is configured to: if the preset vector container is a first preset value, then use the deep learning detection target or the clustering detection target as the associated and fused target; if the preset vector container is a second preset value, then calculate a matching matrix based on the distance between all clustering detection targets and all deep learning detection targets in the preset vector container; when the matching matrix is ​​empty, use the deep learning detection target as the associated and fused target; when the matching matrix is ​​not empty, traverse the matching matrix and use the deep learning detection target and the clustering detection target that meets the preset screening conditions as the associated and fused target. The computing module is also used for: Calculate the distances between all clustered detection targets and all deep learning detection targets in the preset vector container to obtain the Euclidean distance matrix and the IOU distance matrix; The weighted distance matrix is ​​calculated based on the Euclidean distance matrix and the IOU distance matrix, and the matching matrix is ​​obtained by calculating the weighted distance matrix using a preset Hungarian algorithm.

7. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the target association fusion method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the target association fusion method as described in any one of claims 1-5.

Images