Target tracking method and device

By obtaining and matching two-dimensional and three-dimensional target detection information in autonomous driving vehicles and combining prediction information, the problem of difficult sensor information is solved, the accuracy and real-time nature of target tracking is improved, and the real-time perception needs of autonomous driving vehicles are met.

CN119007141APending Publication Date: 2024-11-22BEIJING JINGWEI HIRAIN TECH CO INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310559170.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In existing autonomous vehicles, the target detection information of two-dimensional sensors and three-dimensional sensors is difficult to effectively combine, resulting in limited target tracking accuracy and real-time performance.

Method used

A target tracking method is proposed, by obtaining two-dimensional and three-dimensional target detection information, matching and fusion of information, and using prediction information to predict the target trajectory to achieve accurate tracking of the target. The specific steps include obtaining the target detection information, matching the two-dimensional and three-dimensional information, calculating the interchange ratio, determining the information pair matching situation, and fusion information based on the Kalman filter to determine the target trajectory.

Benefits of technology

It realizes an effective combination of two-dimensional and three-dimensional detection information, improves the accuracy and real-time nature of target tracking, and meets the real-time perception needs of autonomous driving vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007141A_ABST
    Figure CN119007141A_ABST
Patent Text Reader

Abstract

The invention provides a target tracking method and device, and the method comprises the steps: obtaining target detection information at a first moment, which comprises a plurality of pieces of two-dimensional detection information and a plurality of pieces of three-dimensional detection information; matching the two-dimensional detection information with the three-dimensional detection information to obtain a first matching result; under the condition that the first matching result comprises a plurality of first matching information pairs, matching a plurality of pieces of first prediction information with a plurality of pieces of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, the first prediction information is three-dimensional track information of a target at a first moment predicted according to three-dimensional track information of the target at a previous moment of the first moment; and under the condition that the second matching result comprises a plurality of second matching information pairs, fusing two pieces of information contained in the second matching information pairs, and taking a fusion result as three-dimensional track information of the corresponding target at the first moment. According to the invention, two-dimensional detection information and three-dimensional detection information can be combined to realize accurate tracking of the target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a target tracking method and device. Background Art

[0002] In order to realize autonomous driving functions in urban and closed scenarios, autonomous driving vehicles need to accurately perceive obstacles. The main technologies for obstacle perception are target detection and target tracking, that is, obtaining target detection information through target detection technology, and then tracking the target based on the target detection information.

[0003] Current autonomous driving vehicles are generally equipped with two-dimensional sensors (such as cameras) and three-dimensional sensors (such as lidar). Considering that two-dimensional sensors and three-dimensional sensors have their own advantages, the advantages of two-dimensional sensors and three-dimensional sensors can be complemented by each other.

[0004] In order to complement the advantages of two-dimensional sensors and three-dimensional sensors, a current idea is to obtain two-dimensional target detection information based on two-dimensional sensors, obtain three-dimensional target detection information based on three-dimensional sensors, and combine the two-dimensional target detection information with the three-dimensional target detection information to achieve target tracking. How to combine the two-dimensional target detection information with the three-dimensional target detection information to achieve target tracking is a problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the present invention provides a target tracking method and device, which is used to combine two-dimensional target detection information with three-dimensional target detection information to achieve target tracking, and its technical solution is as follows:

[0006] A target tracking method, comprising:

[0007] Acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information;

[0008] Matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result;

[0009] In the case where the first matching result includes a plurality of first matching information pairs, matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment;

[0010] In the case where the second matching result includes several second matching information pairs, two information contained in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each second matching information pair includes matching three-dimensional detection information and first prediction information.

[0011] Optionally, matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result includes:

[0012] Combining the plurality of two-dimensional detection information with the plurality of three-dimensional detection information in pairs to obtain a plurality of information pairs including one two-dimensional detection information and one three-dimensional detection information;

[0013] Determine whether the two-dimensional detection information and the three-dimensional detection information contained in each information pair match;

[0014] Obtaining a first matching result according to a matching condition between the two-dimensional detection information and the three-dimensional detection information included in each information pair;

[0015] The matching of the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result includes:

[0016] Combining the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs in pairs, so as to obtain a plurality of information pairs including a first prediction information and a three-dimensional detection information;

[0017] Determining whether the first prediction information included in each information pair matches the three-dimensional detection information;

[0018] A second matching result is obtained according to the matching conditions between the two-dimensional detection information and the three-dimensional detection information included in each information pair.

[0019] Optionally, the two-dimensional detection information is a two-dimensional target bounding box representing the position of the corresponding target, and the three-dimensional detection information is a three-dimensional target bounding box representing the position of the corresponding target;

[0020] The determining whether the two-dimensional detection information and the three-dimensional detection information included in each information pair match each other includes:

[0021] For each pair of information containing a 3D object bounding box and a 2D object bounding box:

[0022] Projecting the three-dimensional target bounding box in the information pair to the coordinate system where the two-dimensional target bounding box in the information pair is located;

[0023] Calculate the intersection and union ratio of the two-dimensional target bounding box obtained by projection and the two-dimensional target bounding box in the information pair;

[0024] When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair matches the two-dimensional target bounding box. When the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair does not match the two-dimensional target bounding box.

[0025] Optionally, the three-dimensional detection information and the first prediction information are both three-dimensional object bounding boxes representing positions of corresponding objects;

[0026] The determining whether the first prediction information included in each information pair matches the three-dimensional detection information includes:

[0027] For each pair of information containing two 3D object bounding boxes:

[0028] Calculate the intersection-over-union ratio of the top views of the two 3D target bounding boxes contained in the information;

[0029] When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair match; when the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair do not match.

[0030] Optionally, the target tracking method further includes:

[0031] When the first matching result includes some unmatched three-dimensional detection information and the second matching result includes some unmatched first prediction information, matching the some unmatched three-dimensional detection information in the first matching result with the some unmatched first prediction information in the second matching result to obtain a third matching result;

[0032] In the case where the third matching result includes several third matching information pairs, two information contained in each third matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each third matching information pair includes matching three-dimensional detection information and first prediction information.

[0033] Optionally, the target tracking method further includes:

[0034] In a case where the third matching result includes unmatched three-dimensional detection information, determining that a target corresponding to the unmatched three-dimensional detection information in the third matching result is a new target;

[0035] The unmatched three-dimensional detection information in the third matching result is determined as the three-dimensional trajectory information of the corresponding new target at the first moment, and the three-dimensional trajectory information of the corresponding new target at the next moment is predicted based on the unmatched three-dimensional detection information in the third matching result.

[0036] Optionally, the target tracking method further includes:

[0037] In a case where the third matching result includes some unmatched first prediction information, matching the some unmatched first prediction information in the third matching result with some two-dimensional trajectory information at the first moment to obtain a fourth matching result, wherein the some two-dimensional trajectory information at the first moment is determined based on some two-dimensional detection information in the target detection information at the first moment;

[0038] In a case where the fourth matching result includes a fourth matching information pair, the first prediction information in the fourth matching information pair is temporarily determined as the three-dimensional trajectory information of the corresponding target at the first moment.

[0039] Optionally, determining a number of two-dimensional trajectory information at the first moment based on a number of two-dimensional detection information in the target detection information at the first moment includes:

[0040] Matching a plurality of two-dimensional detection information in the target detection information at the first moment with a plurality of second prediction information to obtain a fifth matching result, wherein each second prediction information is two-dimensional trajectory information of a target at the first moment predicted based on the two-dimensional trajectory information of the target at the second moment;

[0041] When the fifth matching result includes several fifth matching information pairs, two information included in each fifth matching information pair are fused, and the fusion result is determined as the two-dimensional trajectory information of the corresponding target at the first moment to obtain several two-dimensional trajectory information at the first moment.

[0042] Optionally, the target tracking method further includes:

[0043] In a case where the fifth matching result includes unmatched two-dimensional detection information, determining the unmatched two-dimensional detection information in the fifth matching result as the two-dimensional trajectory information of the corresponding target at the first moment, and predicting the two-dimensional trajectory information of the corresponding target at the next moment based on the unmatched two-dimensional detection information in the fifth matching result;

[0044] In a case where the fifth matching result includes unmatched second prediction information, the unmatched second prediction information in the fifth matching result is deleted.

[0045] A target tracking device, comprising: a detection information acquisition module, a first information matching module, a second information matching module and a first trajectory information determination module;

[0046] The detection information acquisition module is used to acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information;

[0047] The first information matching module is used to match the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result;

[0048] The second information matching module is used for matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result when the first matching result includes a plurality of first matching information pairs, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on the three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment;

[0049] The first trajectory information determination module is used to fuse two pieces of information included in each second matching information pair when the second matching result includes a plurality of second matching information pairs, and determine the fusion result as the three-dimensional trajectory information of the corresponding target at the first moment, wherein the second matching information pair includes matching three-dimensional detection information and first prediction information.

[0050] The target tracking method and device provided by the present invention first obtains a plurality of two-dimensional detection information at a first moment and a plurality of three-dimensional detection information at a first moment, then matches the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result, and when the first matching result includes a plurality of first matching information pairs, matches a plurality of first prediction information (each first prediction information is the three-dimensional trajectory information of a target at a first moment predicted based on the three-dimensional trajectory information of the target at a second moment) with a plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, and when the second matching result includes a plurality of second matching information pairs, fuses the two pieces of information contained in each second matching information pair, and determines the fusion result as the three-dimensional trajectory information of the corresponding target at the first moment. The target tracking method provided by the present invention can combine the two-dimensional detection information with the three-dimensional detection information to realize accurate tracking of the target, and in addition, the target tracking method provided by the present invention is easy to be implemented in real time on a computing platform, meeting the real-time requirements of an autonomous driving vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A schematic diagram of a target tracking method according to an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of an example of a target tracking method provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of another example of a target tracking method provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of another example of a target tracking method provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the structure of a target tracking device provided by an embodiment of the present invention;

[0057] Figure 6 A schematic diagram of the structure of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In the process of implementing this case, the inventor of this case found that in order to complement the advantages of two-dimensional sensors and three-dimensional sensors, there are currently three ideas, one is multi-sensor data front fusion, the second is feature-level intermediate layer fusion, and the third is target-level post fusion.

[0060] Data-level pre-fusion refers to projecting the raw data of a sensor onto another sensor based on the coordinate transformation relationship. This fusion method can increase the data dimension of a single sensor, but due to the large amount of converted data, the performance requirements of the computing platform are relatively high, and it is difficult to meet the real-time perception requirements of autonomous driving vehicles, and the perception results are greatly affected by the calibration accuracy; feature-level intermediate layer fusion refers to combining the semantic information recognized by a single sensor with the detection process of another sensor, which is usually combined with deep learning technology. Feature fusion will expand the model scale of the deep neural network, the calculation is relatively time-consuming, and it is greatly limited by the accuracy of the main sensor recognition; target-level post-fusion refers to each sensor independently performing target detection, and combining the target detection information of each sensor to achieve target tracking, which belongs to the scope of post-perception processing.

[0061] In view of the large amount of computation required for data-level pre-fusion and feature-level intermediate-layer fusion, the high performance requirements for the computing platform, and the difficulty in meeting the real-time requirements of autonomous driving vehicles, the inventor of this case conducted research along the line of post-fusion at the target level. Through continuous research, he finally proposed a method for combining the two-dimensional target detection information of a two-dimensional sensor with the three-dimensional target detection information of a three-dimensional sensor to achieve target tracking. This method has a small amount of computation and is easy to implement on a computing platform, meeting the real-time requirements of autonomous driving vehicles. Next, the target tracking method provided by the present invention is introduced through the following embodiments.

[0062] See also Figure 1 , which shows a schematic flow chart of a target tracking method provided by an embodiment of the present invention, and may include:

[0063] Step S101: Acquire target detection information at a first moment.

[0064] Among them, the first moment can be the current moment, and the target detection information at the first moment can include some two-dimensional detection information obtained based on a two-dimensional sensor (such as a camera) and some three-dimensional detection information obtained based on a three-dimensional sensor (such as a lidar).

[0065] It should be noted that each two-dimensional detection information corresponds to a target (such as an obstacle), which is two-dimensional information representing the position of the corresponding target, such as a two-dimensional target bounding box representing the position of the corresponding target. Similarly, each three-dimensional detection information corresponds to a target (such as an obstacle), which is three-dimensional information representing the position of the corresponding target, such as a three-dimensional target bounding box representing the position of the corresponding target.

[0066] Step S102: Matching a plurality of two-dimensional detection information with a plurality of three-dimensional detection information in the target detection information at the first moment to obtain a first matching result.

[0067] Optionally, the process of matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information may include:

[0068] Step S1021: Combine a number of two-dimensional detection information and a number of three-dimensional detection information in pairs to obtain a number of information pairs.

[0069] Exemplarily, the target detection information at the first moment includes two two-dimensional detection information 2Ddet1 and 2Ddet2, and three three-dimensional detection information 3Ddet1, 3Ddet2, and 3Ddet3. Combining the two two-dimensional detection information with the three three-dimensional detection information in pairs can obtain information pairs (2Ddet1, 3Ddet1), (2Ddet1, 3Ddet2), (2Ddet1, 3Ddet3), (2Ddet2, 3Ddet1), (2Ddet2, 3Ddet2), and (2Ddet2, 3Ddet3).

[0070] Step S1022: Determine whether the two-dimensional detection information and the three-dimensional detection information included in each information pair match.

[0071] Given that the two-dimensional detection information in the target detection information at the first moment may be a two-dimensional target bounding box representing the position of the corresponding target, and the three-dimensional detection information in the target detection information at the first moment may be a three-dimensional target bounding box representing the position of the corresponding target, whether the two-dimensional detection information and the three-dimensional detection information contained in each information pair match can be determined in the following manner.

[0072] For each pair of information containing 2D detection information and 3D detection information, perform:

[0073] Step S1022-1: Project the three-dimensional detection information in the information pair to the coordinate system where the two-dimensional detection information in the information pair is located.

[0074] Since the three-dimensional detection information in the information pair is a three-dimensional object bounding box, the two-dimensional object bounding box can be obtained by projecting the three-dimensional object bounding box into the coordinate system where the two-dimensional detection information in the information pair is located.

[0075] Step S1022-2: Calculate the intersection-over-union ratio between the two-dimensional detection information obtained by projection and the two-dimensional detection information in the information pair.

[0076] This step calculates the intersection-over-union ratio between the two-dimensional target bounding box obtained by projection and the two-dimensional target bounding box in the information pair.

[0077] The intersection-over-union ratio of two 2D target bounding boxes A and B can be calculated as follows:

[0078]

[0079] Among them, S(A∩B) represents the area of ​​the intersection of the two-dimensional target bounding box A and the two-dimensional target bounding box B, S (A∪B) It represents the area of ​​the union of the two-dimensional target bounding box A and the two-dimensional target bounding box B, that is, the intersection over union (IoU) of the two-dimensional target bounding box A and the two-dimensional target bounding box B is the ratio of the intersection area of ​​the two two-dimensional target bounding boxes to the union area of ​​the two two-dimensional target bounding boxes.

[0080] Step S1022-3: When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the two-dimensional detection information and the three-dimensional detection information contained in the information pair match; when the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the two-dimensional detection information and the three-dimensional detection information contained in the information pair do not match.

[0081] Exemplarily, an information pair is (2Ddet1, 3Ddet1), and the three-dimensional target bounding box 3Ddet1 is projected into the coordinate system where the two-dimensional target bounding box 2Ddet1 is located to obtain the two-dimensional target bounding box, and the intersection and union ratio of the two-dimensional target bounding box obtained by projection and 2Ddet1 is calculated. If the intersection and union ratio is greater than the set intersection and union ratio threshold, it is determined that 2Ddet1 and 3Ddet1 match, otherwise, it is determined that 2Ddet1 and 3Ddet1 do not match. If 2Ddet1 and 3Ddet1 match, it means that the target corresponding to 2Ddet1 and the target corresponding to 3Ddet1 are the same target, that is, both the two-dimensional sensor and the three-dimensional sensor have detected the target.

[0082] Step S1023: Obtain a first matching result according to the matching conditions between the two-dimensional detection information and the three-dimensional detection information included in each information pair.

[0083] Among them, the first matching result obtained according to the matching results corresponding to the plurality of information pairs has the following possibilities:

[0084] First, the first matching result only includes a plurality of matching information pairs. For example, the plurality of two-dimensional detection information includes 2Ddet1 and 2Ddet2, the plurality of three-dimensional detection information includes 3Ddet1 and 3Ddet2, 2Ddet1 matches 3Ddet1, and 2Ddet2 matches 3Ddet2, then the first matching result only includes matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2).

[0085] Second, the first matching result includes a plurality of matching information pairs and a plurality of unmatched three-dimensional detection information. Exemplarily, the plurality of two-dimensional detection information includes 2Ddet1 and 2Ddet2, the plurality of three-dimensional detection information includes 3Ddet1, 3Ddet2, and 3Ddet3, 2Ddet1 matches 3Ddet1, and 2Ddet2 matches 3Ddet2, then the first matching result includes matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2), and unmatched three-dimensional detection information 3Ddet3.

[0086] Third, the first matching result includes a plurality of matching information pairs and unmatched two-dimensional detection information. Exemplarily, the plurality of two-dimensional detection information includes 2Ddet1, 2Ddet2, and 2Ddet3, and the plurality of three-dimensional detection information includes 3Ddet1 and 3Ddet2, 2Ddet1 matches 3Ddet1, and 2Ddet2 matches 3Ddet2, then the first matching result includes matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2), and unmatched two-dimensional detection information 2Ddet3.

[0087] Fourth, the first matching result includes a plurality of matching information pairs, a plurality of unmatched two-dimensional detection information, and a plurality of unmatched three-dimensional detection information. Exemplarily, the plurality of two-dimensional detection information includes 2Ddet1 and 2Ddet2, and the plurality of three-dimensional detection information includes 3Ddet1 and 3Ddet2, and 2Ddet1 matches 3Ddet1, then the first matching result includes a matching information pair (2Ddet1, 3Ddet1), unmatched two-dimensional detection information 2Ddet2, and unmatched three-dimensional detection information 3Ddet2.

[0088] Fifth, the first matching result includes some unmatched two-dimensional detection information and some unmatched three-dimensional detection information. Exemplarily, the some two-dimensional detection information includes 2Ddet1 and 2Ddet2, and the some three-dimensional detection information includes 3Ddet1 and 3Ddet2. 2Ddet1 does not match 3Ddet1 and 3Ddet2, and 2Ddet2 does not match 3Ddet1 and 3Ddet2. Then the first matching result includes the unmatched two-dimensional detection information 2Ddet1 and 2Ddet2, and the unmatched three-dimensional detection information 3Ddet1 and 3Ddet2.

[0089] Step S103: when the first matching result includes a plurality of first matching information pairs, matching a plurality of three-dimensional detection information in the plurality of first matching information pairs with a plurality of first prediction information to obtain a second matching result.

[0090] In order to distinguish from subsequent matching information pairs, in this embodiment, the matching information pair in the first matching result is referred to as the first matching information pair. The first matching information pair includes matching two-dimensional detection information and three-dimensional detection information.

[0091] In this embodiment, each first prediction information is the three-dimensional trajectory information of a target at a first moment predicted based on the three-dimensional trajectory information of the target at a second moment, wherein the second moment is a moment before the first moment. The three-dimensional trajectory information of a target at a certain moment is information representing the position of the target at that moment, for example, the three-dimensional trajectory information of a target at the second moment is a three-dimensional target bounding box representing the position of the target at the second moment.

[0092] Optionally, the process of matching the plurality of three-dimensional detection information in the plurality of first matching information pairs with the plurality of first prediction information may include:

[0093] Step S1031: Combine a number of three-dimensional detection information and a number of first prediction information in a number of first matching information pairs in pairs to obtain a number of information pairs.

[0094] Each information pair obtained through step S1031 includes a three-dimensional detection information and a first prediction information.

[0095] Step S1032: Determine whether the three-dimensional detection information included in each information pair matches the first prediction information.

[0096] Given that both the three-dimensional detection information and the first prediction information can be three-dimensional target bounding boxes that represent the location of the corresponding target, for each information pair obtained through step S1031, the intersection-and-union ratio of the top views (two-dimensional boxes) of the two three-dimensional target bounding boxes contained in the information pair can be calculated. If the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the three-dimensional detection information contained in the information pair matches the first prediction information; otherwise, it is determined that the three-dimensional detection information contained in the information pair does not match the first prediction information.

[0097] Exemplarily, an information pair is (3Ddet1, fused_trk1), and the intersection-and-union ratio of the top view of the three-dimensional detection information 3Ddet (three-dimensional target bounding box) and the top view of the first prediction information fused_trk1 (three-dimensional target bounding box) is calculated. If the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the three-dimensional detection information 3Ddet1 matches the first prediction information fused_trk1, otherwise, it is determined that the three-dimensional detection information 3Ddet1 does not match the first prediction information fused_trk1.

[0098] Step S1033: Obtain a second matching result according to the matching between the three-dimensional detection information included in each information pair and the first prediction information.

[0099] Step S104: when the second matching result includes a plurality of second matching information pairs, two pieces of information included in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment.

[0100] In this embodiment, the matching information pair in the second matching result is referred to as a second matching information pair, and the second matching information pair includes matching two-dimensional detection information and first prediction information.

[0101] Optionally, when fusing two pieces of information included in each second matching information pair, the two pieces of information included in each second matching information pair may be fused based on a three-dimensional Kalman filter.

[0102] The above steps S101 to S104 are described below with reference to an example.

[0103] like Figure 2 As shown, the target detection information at the first moment includes two two-dimensional detection information 2Ddet1 and 2Ddet2, and two three-dimensional detection information 3Ddet1 and 3Ddet2. First, the two two-dimensional detection information is matched with the two three-dimensional detection information. Assuming that 2Ddet1 matches 3Ddet1 and 2Ddet2 matches 3Ddet2, the first matching result includes the first matching information pair (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2). Assuming that the first prediction information includes fused_trk1 and fused_trk2, the two three-dimensional detection information 3Ddet1 and 3Ddet2 in the two first matching information pairs are then matched with the two first prediction information fused_trk1 and fused_trk2. Matching is performed, assuming that 3Ddet1 matches fused_trk1, and 3Ddet2 matches fused_trk2, then the second matching result includes the second matching pair (3Ddet1, fused_trk1) and (3Ddet2, fused_trk2), and finally the two information contained in the second matching pair (3Ddet1, fused_trk1) are fused, and the two information contained in the second matching pair (3Ddet2, fused_trk2) are fused, that is, 3Ddet1 is fused with fused_trk1, and the fusion result is used as the three-dimensional trajectory information of the corresponding target at the first moment, and 3Ddet2 is fused with fused_trk2, and the fusion result is used as the three-dimensional trajectory information of the corresponding target at the first moment.

[0104] The target tracking method provided by the embodiment of the present invention first obtains a number of two-dimensional detection information at a first moment and a number of three-dimensional detection information at a first moment, and then matches the number of two-dimensional detection information with the number of three-dimensional detection information to obtain a first matching result. When the first matching result contains a number of first matching information pairs, a number of first prediction information (the number of first prediction information is the three-dimensional trajectory information of the corresponding target at the first moment predicted based on the three-dimensional trajectory information of the tracked target at the previous moment of the first moment) is matched with the number of three-dimensional detection information in the number of first matching information pairs to obtain a second matching result. When the second matching result contains a number of second matching information pairs, the two pieces of information contained in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment. The target tracking method provided by the embodiment of the present invention can combine the two-dimensional detection information with the three-dimensional detection information to achieve accurate tracking of the target. In addition, the target tracking method provided by the present invention is easy to implement on a computing platform and meets the real-time requirements of an autonomous driving vehicle.

[0105] In another embodiment of the present invention, the target tracking method may further include, in addition to the above steps S101 to S104:

[0106] Step a1: When the first matching result contains some unmatched three-dimensional detection information and the second matching result contains some unmatched first prediction information, match the some unmatched three-dimensional detection information in the first matching result with the some unmatched first prediction information in the second matching result to obtain a third matching result.

[0107] Step a2: when the third matching result includes a plurality of third matching information pairs, two pieces of information included in each third matching information pair are fused, and the fusion result is used as the three-dimensional trajectory information of the corresponding target at the first moment.

[0108] In this embodiment, the matching information pairs in the third matching results are referred to as third matching information pairs, and each third matching information pair includes matching three-dimensional detection information and first prediction information.

[0109] Optionally, when fusing two pieces of information included in each third matching information pair, the two pieces of information included in each third matching information pair may be fused based on a three-dimensional Kalman filter.

[0110] The following describes step S101 to step S104, step a1 and step a2 with reference to an example.

[0111] like Figure 3As shown, the target detection information at the first moment includes two two-dimensional detection information 2Ddet1 and 2Ddet2, and three three-dimensional detection information 3Ddet1, 3Ddet2, and 3Ddet3. The two two-dimensional detection information are matched with the three three-dimensional detection information. It is assumed that the first matching result includes two first matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2), and the unmatched three-dimensional detection information 3Ddet3. Then, the two three-dimensional detection information 3Ddet1 and 3Ddet2 in the two first matching information pairs are matched with the three first prediction information fused_trk1, fused_trk2, and fused_trk3. It is assumed that the second matching result includes two second matching information pairs (3Ddet1, fused_trk1) and (3Ddet2, fused_trk2) and For the unmatched first prediction information fused_trk3, for the two second matching information pairs, 3Ddet1 and fused_trk1 are fused, and the fusion result is used as the three-dimensional trajectory information of the corresponding target at the first moment, 3Ddet2 and fused_trk2 are fused, and the fusion result is used as the three-dimensional trajectory information of the target at the first moment, for the unmatched first prediction information fused_trk3 in the second matching result, it is further matched with the unmatched three-dimensional detection information 3Ddet3 in the first matching result. Assuming that fused_trk3 matches 3Ddet3, the third matching result includes the third matching information pair (fused_trk3, 3Ddet3), fused_trk3 and 3Ddet3 are fused, and the fusion result of fused_trk3 and 3Ddet3 is used as the three-dimensional trajectory information of the corresponding target at the first moment.

[0112] It should be noted that the third matching result may include unmatched 3D detection information. If the third matching result includes unmatched 3D detection information, the target corresponding to the 3D detection information can be determined to be a new target. At this time, the unmatched 3D detection information in the third matching result is determined as the 3D trajectory information of the new target at the first moment, and the 3D trajectory information of the new target at the next moment is predicted based on the unmatched 3D detection information in the third matching result. Optionally, the 3D trajectory information of the new target at the next moment can be predicted based on a 3D Kalman filter and the unmatched 3D detection information in the third matching result is used as the prediction basis.

[0113] Exemplarily, the target detection information at the first moment includes two two-dimensional detection information 2Ddet1 and 2Ddet2, and four three-dimensional detection information 3Ddet1, 3Ddet2, 3Ddet3, and 3Ddet4. It is assumed that the first matching result obtained by matching the two two-dimensional detection information with the four three-dimensional detection information includes two first matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2), and unmatched three-dimensional detection information 3Ddet3 and 3Ddet4. The second matching result is obtained by matching the two three-dimensional detection information 3Ddet1 and 3Ddet2 in the two first matching information pairs with the three first prediction information fused_trk1, fused_trk2, and fused_trk3. Including two second matching information pairs (3Ddet1, fused_trk1) and (3Ddet2, fused_trk2) and the unmatched first prediction information fused_trk3, the third matching result obtained by matching the two unmatched three-dimensional detection information 3Ddet3 and 3Ddet4 in the first matching result with the unmatched first prediction information fused_trk3 in the second matching result includes the third matching information pair (3Ddet3, fused_trk3) and the unmatched three-dimensional detection information 3Ddet4, then it is determined that the target corresponding to 3Ddet4 is the new target, 3Ddet4 is used as the trajectory information of the new target at the first moment, and the three-dimensional trajectory information of the new target at the next moment is predicted based on 3Ddet4.

[0114] In another embodiment of the present invention, the target tracking method may further include, in addition to the above steps S101 to S104 and steps a1 and a2:

[0115] Step b1: when the third matching result includes some unmatched first prediction information, match the some unmatched first prediction information in the third matching result with some two-dimensional trajectory information at the first moment to obtain a fourth matching result.

[0116] Considering that the first prediction information not matched in the third matching result has a higher credibility, in order to accurately track the target, this embodiment further associates the first prediction information not matched in the third matching result with a plurality of two-dimensional trajectory information at the first moment.

[0117] The plurality of two-dimensional trajectory information at the first moment is determined based on the plurality of two-dimensional detection information in the target detection information at the first moment. The specific determination process of the plurality of two-dimensional trajectory information at the first moment will be introduced later.

[0118] Steps S101 to S104 , step a1 , step a2 , and step b1 are described below with reference to an example.

[0119] For example, Figure 4 As shown, the target detection information at the first moment includes two two-dimensional detection information 2Ddet1 and 2Ddet2, and three three-dimensional detection information 3Ddet1, 3Ddet2, and 3Ddet3. The two two-dimensional detection information are matched with the three three-dimensional detection information. Assuming that the first matching result includes two first matching information pairs (2Ddet1, 3Ddet1) and (2Ddet2, 3Ddet2), and the unmatched three-dimensional detection information 3Ddet3, the two three-dimensional detection information 3Ddet1 and 3Ddet2 in the two first matching information pairs are matched with the three first prediction information fused_trk1, fused_trk2, and fused_trk3. Assuming that the second matching result includes two second matching information pairs (3Ddet1, fused_trk1) and (3Ddet2, fused_trk2) and the unmatched first prediction information fused_trk3, for the two second matching information pairs, 3Ddet1 is matched with fused_trk1. Fusion, the fusion result is used as the three-dimensional trajectory information of the corresponding target at the first moment, 3Ddet2 is fused with fused_trk2, and the fusion result is used as the three-dimensional trajectory information of the target at the first moment. For the first prediction information fused_trk3 that is not matched in the second matching result, it is further matched with the three-dimensional detection information 3Ddet3 that is not matched in the first matching result. Assuming that fused_trk3 does not match 3Ddet3, the third matching result obtained includes the unmatched three-dimensional detection information 3Ddet3 and the unmatched first prediction information fused_trk3. For the unmatched three-dimensional detection information 3Ddet3 in the third matching result, its corresponding target is determined as a new target, 3Ddet3 is used as the trajectory information of the new target at the first moment, and the three-dimensional trajectory information of the new target at the next moment is predicted based on 3Ddet3. For the first prediction information fused_trk3 that is not matched in the third matching result, it is matched with several two-dimensional trajectory information at the first moment to obtain a fourth matching result.

[0120] The process of matching the plurality of unmatched first prediction information in the third matching result with the plurality of two-dimensional trajectory information at the first moment to obtain the fourth matching result may include:

[0121] Step b1-1: Combine a number of unmatched first prediction information in the third matching result with a number of two-dimensional trajectory information at the first moment in pairs to obtain a number of information pairs.

[0122] Each information pair includes a first prediction information and a two-dimensional trajectory information.

[0123] Step b1-2: Determine whether the first prediction information included in each information pair matches the two-dimensional trajectory information.

[0124] Given that the first prediction information may be a three-dimensional target bounding box representing the location of the corresponding target, and the two-dimensional trajectory information may be a two-dimensional target bounding box representing the location of the corresponding target, for each information pair including the first prediction information and the two-dimensional trajectory information, whether the two information included in the information pair matches may be determined in the following manner: the first prediction information (three-dimensional target bounding box) in the information pair is projected into a coordinate system where the two-dimensional trajectory information (two-dimensional target bounding box) in the information pair is located; an intersection-and-union ratio of the two-dimensional prediction information (two-dimensional target bounding box) obtained by the projection and the two-dimensional trajectory information (two-dimensional target bounding box) in the information pair is calculated; when the calculated intersection-and-union ratio is greater than a set intersection-and-union ratio threshold, it is determined that the first prediction information included in the information pair matches the two-dimensional trajectory information; and when the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the first prediction information included in the information pair does not match the two-dimensional trajectory information.

[0125] Step b1-3: Obtain a fourth matching result according to the matching conditions between the first prediction information and the two-dimensional trajectory information included in each information pair.

[0126] Step b2: When the fourth matching result includes a fourth matching information pair, the first prediction information in the fourth matching information pair is temporarily determined as the three-dimensional trajectory information of the corresponding target at the first moment.

[0127] For example, Figure 4As shown, the two-dimensional trajectory information at the first moment includes 2Dtrkl and 2Dtrk2, the first prediction information that is not matched in the third matching result is fused_trk3, fused_trk3 is matched with 2Dtrkl and 2Dtrk2 at the first moment, and assuming that fused_trk3 matches 2Dtrkl at the first moment, the fourth matching result includes the fourth matching information pair (fused_trk3, 2Dtrkl) and the two-dimensional trajectory information 2Dtrk2 that is not matched. In view of the fact that the fourth matching result includes the fourth matching information pair (fused_trk3, 2Dtrkl), fused_trk3 in the fourth matching information pair (fused_trk3, 2Dtrkl) is temporarily determined as the three-dimensional trajectory information of the target corresponding to 2Dtrkl at the first moment. Whether fused_trk3 is finally used as the three-dimensional trajectory information of the target corresponding to 2Dtrkl at the first moment is determined according to the target tracking conditions at N consecutive (for example, 3) moments after the first moment. Exemplarily, N is 3, and the three consecutive moments after the first moment are t1, t2, and t3. If a first prediction information corresponding to moment t1 (the three-dimensional trajectory information of the target at moment t1 predicted according to the three-dimensional trajectory information of the target at the first moment) matches 2Dtrkl at moment t1, a first prediction information corresponding to moment t2 (the three-dimensional trajectory information of the target at moment t2 predicted according to the three-dimensional trajectory information of the target at t1) matches 2Dtrkl at moment t2, and a first prediction information corresponding to moment t3 (the three-dimensional trajectory information of the target at moment t3 predicted according to the three-dimensional trajectory information of the target at moment t2) matches 2Dtrkl at moment t3, then fused_trk3 can be determined as the three-dimensional trajectory information of the target corresponding to 2Dtrkl at the first moment at the first moment, otherwise, fused_trk3 is deleted.

[0128] Next, a process of determining a plurality of two-dimensional trajectory information at the first moment based on a plurality of two-dimensional detection information in the target detection information at the first moment is introduced.

[0129] The process of determining a plurality of two-dimensional trajectory information at the first moment based on a plurality of two-dimensional detection information in the target detection information at the first moment may include:

[0130] Step c1: Matching a plurality of two-dimensional detection information in the target detection information at the first moment with a plurality of second prediction information to obtain a fifth matching result.

[0131] Each second prediction information is two-dimensional trajectory information of a target at a first moment predicted based on two-dimensional trajectory information of the target at a second moment (a moment before the first moment).

[0132] The process of matching some two-dimensional detection information in the target detection information at the first moment with some second prediction information to obtain a fifth matching result may include: combining some two-dimensional detection information in the target detection information at the first moment with some second prediction information in pairs to obtain some information pairs containing two-dimensional detection information and second prediction information; determining whether the two-dimensional detection information contained in each information pair matches the second prediction information; and obtaining the fifth matching result based on the matching situation between the two-dimensional detection information contained in each information pair and the second prediction information.

[0133] Given that both the two-dimensional detection information and the two-dimensional trajectory information can be two-dimensional target bounding boxes that characterize the location of the corresponding target, for each information pair containing two-dimensional detection information and second prediction information, the intersection-and-union ratio of the two two-dimensional target bounding boxes contained in the information pair can be calculated. When the calculated intersection-and-union ratio is greater than a set intersection-and-union ratio threshold, it is determined that the two-dimensional detection information contained in the information pair matches the second prediction information. When the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the two-dimensional detection information contained in the information pair does not match the second prediction information.

[0134] Step c2: when the fifth matching result includes a plurality of fifth matching information pairs, two pieces of information included in each fifth matching information pair are fused, and the fusion result is determined as the two-dimensional trajectory information of the corresponding target at the first moment to obtain a plurality of two-dimensional trajectory information.

[0135] Exemplarily, the target detection information at the first moment includes three two-dimensional detection information 2Ddet1, 2Ddet2, and 2Ddet3, the second prediction information includes 2Dtrk1′, 2Dtrk2′, and 2Dtrk3′, 2Ddet1 matches 2Dtrk1′, and 2Ddet2 matches 2Dtrk2′, then the fifth matching result includes the fifth matching information pair (2Ddet1, 2Dtrk1′) and (2Ddet2, 2Dtrk2′), and the unmatched two-dimensional detection information 2Ddet3 and unmatched The matched second prediction information 2Dtrk3′, for the fifth matching information pair (2Ddet1, 2Dtrkl′) and (2Ddet2, 2Dtrk2′), 2Ddet1 and 2Dtrkl′ are fused, and the fusion result is used as the trajectory information 2Dtrk1 of the corresponding target at the first moment, and 2Ddet2 and 2Dtrk2′ are fused, and the fusion result is used as the trajectory information 2Dtrk2 of the corresponding target at the first moment. 2Dtrk1 and 2Dtrk2 are the “several two-dimensional trajectory information at the first moment” in step b1.

[0136] It should be noted that the method of determining the two-dimensional trajectory information of a target at the second moment is similar to the method of determining the two-dimensional trajectory information of a target at the first moment.

[0137] It should also be noted that, in the case where the fifth matching result includes unmatched two-dimensional detection information, the unmatched two-dimensional detection information in the fifth matching result can be used as the two-dimensional trajectory information of the corresponding target at the first moment, and the two-dimensional trajectory information of the corresponding target at the next moment is predicted based on the unmatched two-dimensional detection information in the fifth matching result. Optionally, the two-dimensional trajectory information of the corresponding target at the next moment can be predicted based on a two-dimensional Kalman filter, with the unmatched two-dimensional detection information in the fifth matching result as the prediction basis. In the case where the fifth matching result includes unmatched second prediction information, the unmatched second prediction information in the fifth matching result is deleted.

[0138] An embodiment of the present invention further provides a target tracking device. The target tracking device provided by the embodiment of the present invention is described below. The target tracking device described below and the target tracking method described above can be referred to in correspondence with each other.

[0139] See also Figure 5 , shows a schematic diagram of the structure of a target tracking device provided by an embodiment of the present invention, the target tracking device may include: a detection information acquisition module 501, a first information matching module 502, a second information matching module 503 and a first trajectory information determination module 504.

[0140] The detection information acquisition module 501 is used to acquire target detection information at a first moment.

[0141] The target detection information includes some two-dimensional detection information and some three-dimensional detection information.

[0142] The first information matching module 502 is used to match the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result.

[0143] The second information matching module 503 is used to match the first prediction information with the three-dimensional detection information in the first matching information pairs to obtain a second matching result when the first matching result includes the first matching information pairs.

[0144] Among them, each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, and each first prediction information is the three-dimensional trajectory information of a target at the first moment predicted based on the three-dimensional trajectory information of the target at the second moment, and the second moment is the moment before the first moment.

[0145] The trajectory information determination module 504 is used to fuse the two information included in each second matching information pair when the second matching result includes a plurality of second matching information pairs, and determine the fusion result as the three-dimensional trajectory information of the corresponding target at the first moment, wherein the second matching information pair includes matching three-dimensional detection information and first prediction information.

[0146] Optionally, when the first information matching module 502 matches the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result, it is specifically used to:

[0147] Combining the plurality of two-dimensional detection information with the plurality of three-dimensional detection information in pairs to obtain a plurality of information pairs including one two-dimensional detection information and one three-dimensional detection information;

[0148] Determine whether the two-dimensional detection information and the three-dimensional detection information contained in each information pair match;

[0149] Obtaining a first matching result according to a matching condition between the two-dimensional detection information and the three-dimensional detection information included in each information pair;

[0150] Optionally, when the second information matching module 503 matches the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain the second matching result, it is specifically configured to:

[0151] Combining the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs in pairs, so as to obtain a plurality of information pairs including a first prediction information and a three-dimensional detection information;

[0152] Determining whether the first prediction information included in each information pair matches the three-dimensional detection information;

[0153] A second matching result is obtained according to the matching conditions between the two-dimensional detection information and the three-dimensional detection information included in each information pair.

[0154] Optionally, the two-dimensional detection information is a two-dimensional target bounding box representing the position of the corresponding target, and the three-dimensional detection information is a three-dimensional target bounding box representing the position of the corresponding target.

[0155] When determining whether the two-dimensional detection information and the three-dimensional detection information included in each information pair match, the first information matching module 502 is specifically used to:

[0156] For each pair of information containing a 3D object bounding box and a 2D object bounding box:

[0157] Projecting the three-dimensional target bounding box in the information pair to the coordinate system where the two-dimensional target bounding box in the information pair is located;

[0158] Calculate the intersection and union ratio of the two-dimensional target bounding box obtained by projection and the two-dimensional target bounding box in the information pair;

[0159] When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair matches the two-dimensional target bounding box. When the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair does not match the two-dimensional target bounding box.

[0160] Optionally, the three-dimensional detection information and the first prediction information are both three-dimensional object bounding boxes that represent positions of corresponding objects.

[0161] When determining whether the first prediction information included in each information pair matches the three-dimensional detection information, the second information matching module 503 is specifically used to:

[0162] For each pair of information containing two 3D object bounding boxes:

[0163] Calculate the intersection-over-union ratio of the top views of the two 3D target bounding boxes contained in the information;

[0164] When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair match; when the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair do not match.

[0165] Optionally, the target tracking device may further include: a third information matching module and a second trajectory information determining module.

[0166] The third information matching module is used to match the several unmatched three-dimensional detection information in the first matching result with the several unmatched first prediction information in the second matching result to obtain a third matching result when the first matching result includes several unmatched three-dimensional detection information and the second matching result contains several unmatched first prediction information.

[0167] The second trajectory information determination module is used to fuse two pieces of information included in each third matching information pair when the third matching result includes a plurality of third matching information pairs, and determine the fusion result as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each third matching information pair includes matching three-dimensional detection information and first prediction information.

[0168] Optionally, the target tracking device may further include: an unmatched three-dimensional detection information processing module.

[0169] The unmatched three-dimensional detection information processing module is used to determine, when the third matching result includes unmatched three-dimensional detection information, that the target corresponding to the unmatched three-dimensional detection information in the third matching result is a new target; and determine the unmatched three-dimensional detection information in the third matching result as the three-dimensional trajectory information corresponding to the new target at the first moment, and predict the three-dimensional trajectory information corresponding to the new target at the next moment based on the unmatched three-dimensional detection information in the third matching result.

[0170] Optionally, the target tracking device may further include: a fourth information matching module and a third trajectory information determining module.

[0171] a fourth information matching module, configured to match the plurality of unmatched first prediction information in the third matching result with the plurality of two-dimensional trajectory information at the first moment, when the third matching result includes the plurality of unmatched first prediction information, so as to obtain a fourth matching result, wherein the plurality of two-dimensional trajectory information at the first moment is determined based on the plurality of two-dimensional detection information in the target detection information at the first moment.

[0172] The third trajectory information determination module is configured to temporarily determine the first prediction information in the fourth matching information pair as the three-dimensional trajectory information of the corresponding target at the first moment when the fourth matching result includes a fourth matching information pair.

[0173] Optionally, the target tracking device may further include: a fifth information matching module and a fourth trajectory information determining module.

[0174] a fifth information matching module, configured to match a plurality of two-dimensional detection information in the target detection information at the first moment with a plurality of second prediction information to obtain a fifth matching result, wherein each second prediction information is two-dimensional trajectory information of a target at the first moment predicted based on the two-dimensional trajectory information of the target at the second moment;

[0175] The fourth trajectory information determination module is used to fuse two information included in each fifth matching information pair when the fifth matching result includes a plurality of fifth matching information pairs, and determine the fusion result as the two-dimensional trajectory information of the corresponding target at the first moment, so as to obtain the plurality of two-dimensional trajectory information at the first moment.

[0176] Optionally, the target tracking device may further include: an unmatched two-dimensional detection information processing module and an unmatched two-dimensional prediction information processing module.

[0177] The unmatched two-dimensional detection information processing module is used to determine the unmatched two-dimensional detection information in the fifth matching result as the two-dimensional trajectory information of the corresponding target at the first moment when the fifth matching result includes unmatched two-dimensional detection information, and predict the two-dimensional trajectory information of the corresponding target at the next moment based on the unmatched two-dimensional detection information in the fifth matching result.

[0178] The unmatched two-dimensional prediction information processing module is used to delete the unmatched second prediction information in the fifth matching result when the fifth matching result includes the unmatched second prediction information.

[0179] The target tracking device provided by the embodiment of the present invention can combine two-dimensional detection information with three-dimensional detection information to achieve accurate tracking of the target. In addition, the target tracking device provided by the present invention meets the real-time requirements of autonomous driving vehicles.

[0180] The present invention also provides a processing device, see Figure 6 , shows a schematic diagram of the structure of the processing device, which may include: a processor 601, a communication interface 602, a memory 603 and a communication bus 604;

[0181] In the embodiment of the present invention, the number of the processor 601, the communication interface 602, the memory 603, and the communication bus 604 is at least one, and the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604;

[0182] The processor 601 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0183] The memory 603 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0184] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0185] Acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information;

[0186] Matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result;

[0187] In the case where the first matching result includes a plurality of first matching information pairs, matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment;

[0188] In the case where the second matching result includes several second matching information pairs, two information contained in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each second matching information pair includes matching three-dimensional detection information and first prediction information.

[0189] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0190] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0191] Acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information;

[0192] Matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result;

[0193] In the case where the first matching result includes a plurality of first matching information pairs, matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment;

[0194] In the case where the second matching result includes several second matching information pairs, two information contained in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each second matching information pair includes matching three-dimensional detection information and first prediction information.

[0195] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0196] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0197] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0198] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method, characterized in that: include: Acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information; Matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result; In the case where the first matching result includes a plurality of first matching information pairs, matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment; In the case where the second matching result includes several second matching information pairs, two information contained in each second matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each second matching information pair includes matching three-dimensional detection information and first prediction information.

2. The target tracking method according to claim 1, characterized in that: The matching the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result includes: Combining the plurality of two-dimensional detection information with the plurality of three-dimensional detection information in pairs to obtain a plurality of information pairs including one two-dimensional detection information and one three-dimensional detection information; Determine whether the two-dimensional detection information and the three-dimensional detection information contained in each information pair match; Obtaining a first matching result according to a matching condition between the two-dimensional detection information and the three-dimensional detection information included in each information pair; The matching of the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result includes: Combining the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs in pairs, so as to obtain a plurality of information pairs including a first prediction information and a three-dimensional detection information; Determining whether the first prediction information included in each information pair matches the three-dimensional detection information; A second matching result is obtained according to the matching conditions between the two-dimensional detection information and the three-dimensional detection information included in each information pair.

3. The target tracking method according to claim 2, characterized in that: The two-dimensional detection information is a two-dimensional target bounding box representing the location of the corresponding target, and the three-dimensional detection information is a three-dimensional target bounding box representing the location of the corresponding target; The determining whether the two-dimensional detection information and the three-dimensional detection information included in each information pair match each other includes: For each pair of information containing a 3D object bounding box and a 2D object bounding box: Projecting the three-dimensional target bounding box in the information pair to the coordinate system where the two-dimensional target bounding box in the information pair is located; Calculate the intersection and union ratio of the two-dimensional target bounding box obtained by projection and the two-dimensional target bounding box in the information pair; When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair matches the two-dimensional target bounding box. When the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the three-dimensional target bounding box contained in the information pair does not match the two-dimensional target bounding box.

4. The target tracking method according to claim 2, characterized in that: The three-dimensional detection information and the first prediction information are both three-dimensional object bounding boxes representing the positions of the corresponding objects; The determining whether the first prediction information included in each information pair matches the three-dimensional detection information includes: For each pair of information containing two 3D object bounding boxes: Calculate the intersection-over-union ratio of the top views of the two 3D target bounding boxes contained in the information; When the calculated intersection-and-union ratio is greater than the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair match; when the calculated intersection-and-union ratio is less than or equal to the set intersection-and-union ratio threshold, it is determined that the two three-dimensional target bounding boxes contained in the information pair do not match.

5. The target tracking method according to any one of claims 1 to 4, characterized in that: Also includes: When the first matching result includes some unmatched three-dimensional detection information and the second matching result includes some unmatched first prediction information, matching the some unmatched three-dimensional detection information in the first matching result with the some unmatched first prediction information in the second matching result to obtain a third matching result; In the case where the third matching result includes several third matching information pairs, two information contained in each third matching information pair are fused, and the fusion result is determined as the three-dimensional trajectory information of the corresponding target at the first moment, wherein each third matching information pair includes matching three-dimensional detection information and first prediction information.

6. The target tracking method according to claim 5, characterized in that: Also includes: In a case where the third matching result includes unmatched three-dimensional detection information, determining that a target corresponding to the unmatched three-dimensional detection information in the third matching result is a new target; The unmatched three-dimensional detection information in the third matching result is determined as the three-dimensional trajectory information of the corresponding new target at the first moment, and the three-dimensional trajectory information of the corresponding new target at the next moment is predicted based on the unmatched three-dimensional detection information in the third matching result.

7. The target tracking method according to claim 5, characterized in that: Also includes: In a case where the third matching result includes some unmatched first prediction information, matching the some unmatched first prediction information in the third matching result with some two-dimensional trajectory information at the first moment to obtain a fourth matching result, wherein the some two-dimensional trajectory information at the first moment is determined based on some two-dimensional detection information in the target detection information at the first moment; In a case where the fourth matching result includes a fourth matching information pair, the first prediction information in the fourth matching information pair is temporarily determined as the three-dimensional trajectory information of the corresponding target at the first moment.

8. The target tracking method according to claim 7, characterized in that: Determining a plurality of two-dimensional trajectory information at the first moment based on a plurality of two-dimensional detection information in the target detection information at the first moment includes: Matching a plurality of two-dimensional detection information in the target detection information at the first moment with a plurality of second prediction information to obtain a fifth matching result, wherein each second prediction information is two-dimensional trajectory information of a target at the first moment predicted based on the two-dimensional trajectory information of the target at the second moment; When the fifth matching result includes several fifth matching information pairs, two information included in each fifth matching information pair are fused, and the fusion result is determined as the two-dimensional trajectory information of the corresponding target at the first moment to obtain several two-dimensional trajectory information at the first moment.

9. The target tracking method according to claim 8, characterized in that: Also includes: In a case where the fifth matching result includes unmatched two-dimensional detection information, determining the unmatched two-dimensional detection information in the fifth matching result as the two-dimensional trajectory information of the corresponding target at the first moment, and predicting the two-dimensional trajectory information of the corresponding target at the next moment based on the unmatched two-dimensional detection information in the fifth matching result; In a case where the fifth matching result includes unmatched second prediction information, the unmatched second prediction information in the fifth matching result is deleted.

10. A target tracking device, characterized in that: include: A detection information acquisition module, a first information matching module, a second information matching module and a first trajectory information determination module; The detection information acquisition module is used to acquire target detection information at a first moment, wherein the target detection information includes a plurality of two-dimensional detection information and a plurality of three-dimensional detection information; The first information matching module is used to match the plurality of two-dimensional detection information with the plurality of three-dimensional detection information to obtain a first matching result; The second information matching module is used for matching the plurality of first prediction information with the plurality of three-dimensional detection information in the plurality of first matching information pairs to obtain a second matching result when the first matching result includes a plurality of first matching information pairs, wherein each first matching information pair includes matching two-dimensional detection information and three-dimensional detection information, each first prediction information is three-dimensional trajectory information of a target at the first moment predicted based on the three-dimensional trajectory information of the target at the second moment, and the second moment is a moment before the first moment; The first trajectory information determination module is used to fuse two pieces of information included in each second matching information pair when the second matching result includes a plurality of second matching information pairs, and determine the fusion result as the three-dimensional trajectory information of the corresponding target at the first moment, wherein the second matching information pair includes matching three-dimensional detection information and first prediction information.