A fusion method for tracking targets using visual cameras and millimeter-wave radar
The multi-frame timing information of visual cameras and millimeter-wave radar and the neural network model are linked to the target, which solves the correlation errors and large amount of calculations of multi-sensor fusion in autonomous driving, and achieves efficient and robust target detection and fusion effects.
Patent Information
- Application Number
- CN202411088700.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing multi-sensor tracking target fusion is prone to association errors, requiring a large number of rules and thresholds based on manual design, and the large amount of calculation cannot meet the needs of the real-time system of autonomous driving.
Multi-frame timing information of visual cameras and millimeter wave radars are used to relate the target, and data-driven target correlation is used to relate the data without relying on prior knowledge. The state vector is updated through the Kalman filter to reduce manual strategies and threshold settings. .
It improves the fault tolerance rate of single sensor detection results and the overall accuracy/recall rate of the system, maintains robust performance under harsh operating conditions, fast training convergence and fast inference speed, and is suitable for embedded platform deployment.
Smart Images

Figure CN118628881B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for fusing a visual camera and a millimeter-wave radar to track a target. Background Art
[0002] Autonomous driving technology is a systematic project, which mainly consists of three modules: environmental perception, planning and decision-making, and control execution. Environmental perception is a module that senses the surrounding road environment through sensors such as cameras, millimeter-wave radars, and lidars to obtain information such as the category, location, size, and speed of surrounding targets. Accurate and robust target detection in complex scenarios is one of the challenges currently faced by high-level autonomous driving technology.
[0003] Visual cameras have the advantages of large field of view, low cost, high resolution, and rich perception information, but they have poor adaptability to light and weather, and cannot accurately obtain distance measurement and speed measurement information. Millimeter-wave radar is a sensor that measures distance and speed based on electromagnetic signals reflected by the target. It has the advantages of full-condition operation and high-precision distance measurement and speed measurement. However, it has the disadvantages of difficulty in obtaining obstacle type, size, and orientation information, metal sensitivity, and difficulty in distinguishing relatively stationary targets. In summary, cameras and millimeter-wave radar sensors each have their own advantages and limitations, and their perception capabilities can complement each other; the perception algorithm based on the fusion of the two can significantly improve environmental perception capabilities and can better cope with autonomous driving tasks in complex scenarios.
[0004] The task of obstacle fusion is to associate and fuse the tracking target results output by multiple sensors to obtain perception results with better indicators. High-precision fusion is based on correct target association, so target association is the core content of obstacle fusion algorithm. There are four mainstream target association algorithms: 1. Decision-based nearest neighbor algorithm; 2. Multi-hypothesis tracking algorithm based on probability statistics; 3. Probability / joint probability data association algorithm based on Bayesian formula; 4. Fuzzy data association algorithm based on fuzzy logic and fuzzy clustering.
[0005] The above algorithms may have the following problems: 1. The obstacle fusion scheme based on single-frame sensor information has low robustness, is sensitive to noise and clutter, and is prone to association errors when targets are dense; 2. The performance depends on prior knowledge of targets and clutter, and requires a large number of manually designed rules and thresholds; 3. The computational complexity is large and cannot meet the needs of real-time autonomous driving systems. Summary of the invention
[0006] The present invention provides a method for fusing visual cameras and millimeter-wave radar target tracking to solve the technical problems that the existing multi-sensor target tracking fusion is prone to association errors, requires a large number of manually designed rules and thresholds, and has a large amount of calculation that cannot meet the needs of autonomous driving real-time systems.
[0007] One aspect of the present invention is to provide a method for fusing a visual camera and a millimeter-wave radar to track a target, the method comprising the following steps:
[0008] S1. Get the first frame sensor message and generate the first frame visual camera tracking target state vector set Or the state vector set of the millimeter wave radar tracking target , initialize the latest state vector queue set of the visual camera tracking target in the first frame Initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target , initialize the set of visual camera-millimeter wave radar fusion tracking targets in the first frame ;
[0009] S2. Get the t Frame sensor message, generating the t Frame vision camera tracking target state vector collection Or the state vector set of the millimeter wave radar tracking target , update t Frame vision camera tracking target latest state vector queue collection , update t Frame millimeter wave radar tracking target latest state vector queue collection ;
[0010] S3. t Each visual camera tracks the target with the t Each millimeter-wave radar tracking target is paired in each frame, and a time-series visual camera-millimeter-wave radar fusion state vector is constructed for each target pair. , input visual camera-millimeter wave radar tracking target timing matching classification network, output the matching type and matching probability of each target pair, and generate the first t Frame vision camera-millimeter wave radar fusion tracking target matching pair set ;
[0011] S4. Update t A collection of frame vision camera-millimeter wave radar fusion tracking targets ;
[0012] S5. Get the t+1 Frame sensor message, repeat steps S2 to S4.
[0013] In a preferred embodiment, in step S1,
[0014] If the first frame sensor message is a visual camera tracking target message:
[0015] Generate the first frame visual camera tracking target state vector set ;
[0016] in, Indicates the first frame i A visual camera tracks the target ID; Indicates the first frame i A visual camera tracks the target state vector; Indicates the number of targets tracked by the visual camera in the first frame;
[0017] Initialize the latest state vector queue set of the visual camera tracking target in the first frame ;
[0018] At the same time, initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target to an empty set , and initialize the set of visual camera-millimeter wave radar fusion tracking targets in the first frame ;
[0019] in, ; Indicates the length is T Frame 1 of i A queue of the latest state vectors of the visual camera tracking target;
[0020] Indicates the first frame i The state vector of the target tracked by a visual camera; Indicates the first frame i A visual camera tracks the target ID; Indicates the number of targets tracked by the visual camera in the first frame;
[0021] If the first frame of sensor message is a millimeter-wave radar tracking target message:
[0022] Generate the first frame of millimeter wave radar tracking target state vector set ;
[0023] in, Indicates the first frame i Millimeter wave radar tracking target ID; Indicates the first frame i A millimeter-wave radar tracking target state vector; Indicates the number of targets tracked by the millimeter-wave radar in the first frame;
[0024] Initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target ;
[0025] At the same time, initialize the latest state vector queue set of the visual camera tracking target in the first frame to an empty set , and initialize the first frame visual camera-millimeter wave radar fusion tracking target set to an empty set ;
[0026] in, ; Indicates the length is T Frame 1 of i A queue of the latest state vectors of the millimeter-wave radar tracking target;
[0027] Indicates the first frame i A millimeter-wave radar tracking target state vector; Indicates the first frame i Millimeter wave radar tracking target ID; Indicates the number of targets tracked by the millimeter-wave radar in the first frame.
[0028] In a preferred embodiment, in step S2,
[0029] Jordi t The frame sensor message is a visual camera tracking target message:
[0030] The generated t Frame vision camera tracking target state vector collection ;
[0031] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. i A visual camera tracks the target state vector; Indicates t The number of targets tracked by the frame vision camera;
[0032] Use t Frame vision camera tracking target state vector collection Update t Frame vision camera tracking target latest state vector queue collection :
[0033] Initialize t Frame vision camera tracking target latest state vector queue collection = ;
[0034] Traverse t The state vector set of the frame vision camera tracking target All visual cameras in the track target;
[0035] Jordi tFrame vision camera tracking target latest state vector queue collection There is t Frame No. i A queue of the latest state vectors corresponding to the target tracked by a visual camera;
[0036] Then the t Frame No. i The visual camera tracks the target state vector , increase to t Frame No. i The length of the target tracked by a visual camera is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target , and delete the queue with length T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The head element of the team;
[0037] Jordi t Frame vision camera tracking target latest state vector queue collection There is no t Frame No. i A queue of the latest state vectors corresponding to the target tracked by a visual camera;
[0038] The new length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target , and the newly created length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target Join t Frame vision camera tracking target latest state vector queue collection middle;
[0039] in, ;
[0040] Indicates that the length of the new T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i A visual camera tracks the target state vector;
[0041] After the traversal is completed, tFrame vision camera tracking target latest state vector queue collection The latest state vector queue element that is not updated in is deleted;
[0042] Update the following method t Frame millimeter wave radar tracking target latest state vector queue collection :
[0043] Initialize t The latest state vector queue collection of the frame millimeter wave radar tracking target = ;
[0044] The first t Frame millimeter wave radar tracking target latest state vector queue collection In the example, each length is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The head element of the team is deleted;
[0045] At the same time, each length is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The tail element of the queue is copied and added to the tail of the queue, or in each length T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The tail element of the queue is supplemented with a zero vector;
[0046] Jordi t The frame sensor message is a millimeter wave radar tracking target message:
[0047] Generate t Frame millimeter wave radar tracking target state vector set ;
[0048] in, Indicates t Frame No. i Millimeter wave radar tracking target ID; Indicates t Frame No. i A millimeter-wave radar tracking target state vector; Indicates t Number of targets tracked by frame millimeter wave radar;
[0049] Use t Frame millimeter wave radar tracking target state vector set Updatet Frame millimeter wave radar tracking target latest state vector queue collection :
[0050] Initialize t The latest state vector queue collection of the frame millimeter wave radar tracking target = ;
[0051] Traverse t Frame millimeter wave radar tracking target state vector set All millimeter-wave radars track targets;
[0052] Jordi t Frame millimeter wave radar tracking target latest state vector queue collection There is t Frame No. i The latest state vector queue corresponding to the millimeter-wave radar tracking target;
[0053] Then the t Frame No. i The state vector of the target tracked by the millimeter-wave radar , increase to t Frame No. i The length corresponding to a millimeter-wave radar tracking target is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target , and delete the queue with length T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The head element of the team;
[0054] Jordi t Frame millimeter wave radar tracking target latest state vector queue collection There is no t Frame No. i The latest state vector queue corresponding to the millimeter-wave radar tracking target;
[0055] The new length is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target , and the newly created length is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target Join t Frame millimeter wave radar tracking target latest state vector queue collection middle;
[0056] in, ;
[0057] Indicates that the length of the new T No. t Frame No. i A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicates t Frame No. i A state vector of a millimeter-wave radar tracking target;
[0058] After the traversal is completed, t Frame millimeter wave radar tracking target latest state vector queue collection The latest state vector queue element that is not updated in is deleted;
[0059] Update the following method t Frame vision camera tracking target latest state vector queue collection :
[0060] Initialize t The latest state vector queue collection of the frame vision camera tracking target = ;
[0061] The first t Frame vision camera tracking target latest state vector queue collection In the example, each length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The head element of the team is deleted;
[0062] At the same time, each length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The tail element of the queue is copied and added to the tail of the queue, or in each length T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The tail element of the queue is padded with a zero vector.
[0063] In a preferred embodiment, in step S3, t Each visual camera tracks the target with the t Each millimeter-wave radar tracking target in the frame is paired with each other to construct the tFrame vision camera-millimeter wave radar fusion tracking target pair collection , expressed as follows:
[0064] ;
[0065] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. j Millimeter wave radar tracking target ID; Indicates t The number of targets tracked by the frame vision camera; Indicates t Number of targets tracked by frame millimeter wave radar;
[0066] Traverse the constructed t Visual camera-millimeter wave radar fusion tracking target pair collection Each visual camera-millimeter wave radar fusion target pair in , the corresponding time-series visual camera-millimeter wave radar fusion state vector is constructed by the following method :
[0067] From t Frame vision camera tracking target latest state vector queue collection Extract the t Frame No. i A queue of the latest state vectors of the visual camera tracking target ;
[0068] From t Frame millimeter wave radar tracking target latest state vector queue collection Extract the t Frame No. j The latest state vector queue of the millimeter wave radar tracking target ;
[0069] in, Indicates the extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i The latest state vector of the visual camera tracking target;
[0070] Indicates the extracted t Frame No. j A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicatest Frame No. j The latest state vector of a millimeter-wave radar tracking target;
[0071] T Indicates the extracted t Frame No. i The length of the queue of the latest state vector of the visual camera tracking target / the extracted t Frame No. j The length of the latest state vector queue of a millimeter-wave radar tracking target;
[0072] The extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target Each element in t Frame No. j The latest state vector queue of the millimeter wave radar tracking target Each element in is normalized by mean-variance and then concatenated to obtain the constructed time-series visual camera-millimeter wave radar fusion state vector:
[0073] ;
[0074] in, and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ;
[0075] Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. j Millimeter wave radar tracking target ID;
[0076] The constructed t Visual camera-millimeter wave radar fusion tracking target pair collection Each visual camera-millimeter wave radar fusion target pair in Corresponding time-series visual camera-millimeter wave radar fusion state vector Input visual camera-millimeter wave radar time series tracking target matching classification network, output each target pair Match type and match probability , construct the t Frame vision camera-millimeter wave radar fusion target pair matching result set ;
[0077] in, For visual camera-millimeter wave radar fusion target pair The match type, Indicates mismatch. Indicates matching, represents the matching probability;
[0078] Traverse t Frame vision camera-millimeter wave radar fusion target pair matching result set Each target pair The matching results are:
[0079] For t Frame Each visual camera tracks the target and selects the matching type , matching probability Largest millimeter wave radar tracking target As a millimeter wave radar to match and track targets;
[0080] If the matching probability If it is less than the preset threshold, t Frame No. i The millimeter wave radar fusion matching tracking target Set to 0;
[0081] Build t Frame vision camera-millimeter wave radar fusion matching tracking target matching pair set Expressed in the following way:
[0082] ;
[0083] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. i The ID of the millimeter-wave radar tracking target matched with the visual camera tracking target; Indicates t The number of objects tracked by the frame vision camera.
[0084] In a preferred embodiment, in step S3, the visual camera-millimeter wave radar temporal tracking target matching classification network used is a pre-trained deep learning neural network;
[0085] The visual camera-millimeter wave radar time series tracking target matching classification network is trained using manually annotated data. The dataset contains multiple visual camera-millimeter wave radar fusion target pairs. Corresponding time-series visual camera-millimeter wave radar fusion state vector Fusion of multiple visual cameras and millimeter wave radars for target alignment The match type.
[0086] In a preferred embodiment, in step S5, the first t Frame vision camera-millimeter wave radar fusion tracking target collection :
[0087] Initialize t A collection of frame vision camera-millimeter wave radar fusion tracking targets = ;
[0088] Traversing the set of target matching pairs for visual camera-millimeter wave radar fusion tracking Each visual camera-millimeter wave radar fusion tracking target matching pair ;
[0089] When t A collection of frame vision camera-millimeter wave radar fusion tracking targets In t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding t-1 Frame No. k A visual camera tracks the target to meet hour;
[0090] Jordi t If the frame sensor message is a visual camera tracking target message, the t Frame No. i The visual camera tracks the target state vector As the t Frame No. k The observation vector of the target tracked by a visual camera-millimeter wave radar fusion ;
[0091] Jordi t If the frame sensor message is a millimeter wave radar tracking target message, the t Frame No. i The state vector of the target tracked by the millimeter-wave radar As the t Frame No. k Target observation vector for visual camera-millimeter wave radar fusion tracking ;
[0092] in, ;
[0093] Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target ; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracking target ID; Indicates t- 1 Frame No. k A visual camera-millimeter wave radar fusion tracking target Corresponding millimeter-wave radar tracking target ID; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The state vector of
[0094] The first t Frame No. k The observation vector of the target tracked by a visual camera-millimeter wave radar fusion With t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The state vector , and input them into the Kalman filter together, so that t-1 State vector of target tracked by frame vision camera-millimeter wave radar fusion Update to t State vector of target tracked by frame vision camera-millimeter wave radar fusion ;
[0095] At the same time, the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding millimeter wave radar tracks the target Updated to t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. , generating t Frame No. k A visual camera-millimeter wave radar fusion tracking target ;
[0096] in, ;
[0097] Indicates t Frame No. k A visual camera-millimeter wave radar fusion tracks the target; Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. i The ID of the millimeter-wave radar tracking target matched with the visual camera tracking target; Indicates t The state vector of the frame vision camera-millimeter wave radar fusion tracking target;
[0098] The first t Frame vision camera-millimeter wave radar fusion tracking target collection Unupdated visual camera-millimeter wave radar fusion tracking target Delete, complete t A collection of frame vision camera-millimeter wave radar fusion tracking targets Updates;
[0099] When t A collection of frame vision camera-millimeter wave radar fusion tracking targets There is no t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracks the target satisfy hour;
[0100] Jordi t The frame sensor message is a visual camera tracking target message, and it will be t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. The corresponding state vector As the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0101] The first t Frame No. i The visual camera tracks the target state vector As the t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0102] Jordi t The frame sensor is a millimeter wave radar tracking target message, then the first t Frame No. i The visual camera tracks the target state vector As the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0103] Will be with t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. The corresponding state vector As the t Frame vision camera-millimeter wave radar fusion tracking target The observation vector ;
[0104] The first t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector and t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector , sequentially input into the Kalman filter, update the t Frame No. k The state vector of the target tracked by the visual camera-millimeter wave radar fusion ;
[0105] Initialize t Frame No. k A visual camera-millimeter wave radar fusion tracking target , will t Frame No. k A visual camera-millimeter wave radar fusion tracking target Join t A collection of frame vision camera-millimeter wave radar fusion tracking targets and t A collection of frame vision camera-millimeter wave radar fusion tracking targets Unupdated visual camera-millimeter wave radar fusion tracking target Delete, complete t A collection of frame vision camera-millimeter wave radar fusion tracking targets Updates.
[0106] Compared with the prior art, the present invention has the following beneficial effects:
[0107] The present invention provides a method for fusing visual cameras and millimeter-wave radars to track targets. The method uses multi-frame timing information of sensors to associate multi-sensor targets (obstacles), thereby improving the fault tolerance of single sensor detection and tracking results and the overall precision / recall rate of the system, and maintaining robust performance under various harsh working conditions (such as single sensor failure and frame loss).
[0108] The present invention provides a fusion method for tracking targets with a visual camera and a millimeter-wave radar, which uses a data-driven neural network model to associate targets (obstacles). It does not rely on any prior knowledge of targets and clutter, and the model parameters are completely based on data-driven training, avoiding a large number of manual strategies and threshold settings.
[0109] The present invention provides a method for integrating a visual camera and a millimeter-wave radar to track a target, which has fast training convergence and reasoning speed, and can be quickly deployed and run in real time on an embedded platform, thus having direct application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0110] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0111] Figure 1 The present invention is a flow chart of a method for integrating a visual camera and a millimeter-wave radar to track a target. DETAILED DESCRIPTION
[0112] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.
[0113] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0114] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0115] Combination Figure 1 According to an embodiment of the present invention, a method for integrating a visual camera and a millimeter-wave radar to track a target is provided, comprising the following method steps:
[0116] Step S1: Get the first frame sensor message and generate the first frame visual camera tracking target state vector set Or the state vector set of the millimeter wave radar tracking target , initialize the latest state vector queue set of the visual camera tracking target in the first frame Initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target Initialize the first frame of visual camera-millimeter wave radar fusion tracking target set .
[0117] In this embodiment, a visual camera and a millimeter-wave radar are installed on the vehicle, and the visual camera is used to capture multiple frames of targets (obstacles) tracked by the visual camera (such as image targets), and the millimeter-wave radar is used to collect multiple frames of targets (obstacles) tracked by the millimeter-wave radar (such as point cloud data).
[0118] Each frame of the visual camera tracks multiple targets, and each frame of the millimeter-wave radar tracks multiple targets.
[0119] The sensor message comes from a target (obstacle) tracked by a multi-frame visual camera (such as an image target) captured by a visual camera or a target (obstacle) tracked by a multi-frame millimeter-wave radar (such as point cloud data) collected by a millimeter-wave radar.
[0120] The visual camera tracking target of the present invention refers to the target tracked by the visual camera. The millimeter wave radar tracking target refers to the target tracked by the millimeter wave radar. The visual camera-millimeter wave radar fusion tracking target refers to the target that needs to be fused with the target tracked by the visual camera and the target tracked by the millimeter wave radar.
[0121] According to an embodiment of the present invention, if the first frame sensor message is a visual camera tracking target message:
[0122] Generate the first frame visual camera tracking target state vector set ;
[0123] in, Indicates the first frame i A visual camera tracks the target ID; Indicates the first frame i A visual camera tracks the target state vector; Indicates the number of targets tracked by the visual camera in the first frame.
[0124] The state vector of the present invention is used for matching the visual camera-millimeter wave radar tracking target, and the elements of the state vector can be freely selected according to the actual situation.
[0125] In one embodiment, the first frame i The visual camera tracks the target state vector Expressed in the following way:
[0126] ;
[0127] in, Indicates the first frame i A visual camera tracks the longitudinal coordinate of the target in the vehicle coordinate system; Indicates the first frame i A visual camera tracks the lateral coordinates of the target in the vehicle coordinate system; Indicates the first frame i A visual camera tracks the longitudinal velocity of the target in the vehicle coordinate system; Indicates the first frame i A visual camera tracks the lateral velocity of the target in the vehicle coordinate system; Indicates the first frame i The semantic type of the target tracked by a visual camera.
[0128] Initialize the latest state vector queue set of the visual camera tracking target in the first frame .
[0129] At the same time, initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target to an empty set , and initialize the first frame of visual camera-millimeter wave radar fusion tracking target set .
[0130] in, ; Indicates the length is T Frame 1 of i The latest state vector queue of the visual camera tracking target; the queue element is i The latest state vector of a visual camera tracking the target from the past frame to the current frame.
[0131] Indicates the first frame i A visual camera tracks the target state vector; Indicates the first frame i A visual camera tracks the target ID; Indicates the number of targets tracked by the visual camera in the first frame.
[0132] In the present invention, when a certain frame (for example, the first frame) of the visual camera tracking target does not exist / is lost / missed i A visual camera tracks the target, then the length is T Frame 1 of i The latest state vector queue of the visual camera tracking target It is padded with an all-zero vector of the same dimension as the state vector of the visual camera tracking target.
[0133] For example, in this embodiment, ;in, The dimension of the state vector representing the target being tracked by the visual camera. Preferably, =5.
[0134] The set of each frame of visual camera-millimeter wave radar fusion tracking target of the present invention includes a visual camera tracking target, a millimeter wave radar tracking target (if it does not exist, it is filled with 0), and a state vector of the visual camera-millimeter wave radar fusion tracking target.
[0135] In this embodiment, the first frame visual camera-millimeter wave radar fusion tracking target is the target tracked by the visual camera. In the 1st frame, i A visual camera tracks the target (with the first frame as the i ID representation of the target tracked by a visual camera), the first frame i The millimeter-wave radar tracking target is filled with 0, and the state vector of the visual camera-millimeter-wave radar fusion tracking target in the first frame is i The state vector of the visual camera tracking the target .
[0136] According to an embodiment of the present invention, if the first frame sensor message is a millimeter wave radar tracking target message:
[0137] Generate the first frame of millimeter wave radar tracking target state vector set ;
[0138] in, Indicates the first frame i Millimeter wave radar tracking target ID; Indicates the first frame i A millimeter-wave radar tracking target state vector; Indicates the number of targets tracked by the millimeter-wave radar in the first frame.
[0139] In one embodiment, the first frame i The state vector of the target tracked by the millimeter-wave radar Expressed in the following way:
[0140] ;
[0141] in, Indicates the first frame i The longitudinal coordinate of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates the first frame i The lateral coordinates of the target tracked by a millimeter-wave radar in the vehicle coordinate system; Indicates the first frame i A millimeter-wave radar tracks the longitudinal velocity of the target in the vehicle coordinate system; Indicates the first frame i A millimeter-wave radar tracks the lateral velocity of the target in the vehicle coordinate system; Indicates the first frame i The longitudinal acceleration of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates the first frame i The lateral acceleration of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates the first frame i The type of lane line where the millimeter-wave radar tracks the target.
[0142] Initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target ;
[0143] At the same time, initialize the latest state vector queue set of the visual camera tracking target in the first frame to an empty set , and initialize the first frame visual camera-millimeter wave radar fusion tracking target set to an empty set ;
[0144] in, ; Indicates the length is T Frame 1 of i The latest state vector queue of the millimeter-wave radar tracking target; the queue element is i The latest state vector of a millimeter-wave radar tracking target from the past frame to the current frame.
[0145] Indicates the first frame i A state vector of a millimeter-wave radar tracking target; Indicates the first frame i Millimeter wave radar tracking target ID; Indicates the number of targets tracked by the millimeter-wave radar in the first frame.
[0146] When the target is not present / lost / missed in a certain frame (e.g., the first frame) of the millimeter-wave radar tracking target i A millimeter-wave radar tracks the target, then the length is T Frame 1 of i The latest state vector queue of the target tracked by the millimeter-wave radar It is filled with a full zero vector with the same dimension as the state vector of the millimeter wave radar tracking target.
[0147] For example, in this embodiment, ;in, The dimension of the state vector representing the target being tracked by the millimeter wave radar. Preferably, =7.
[0148] Step S2: Get t Frame sensor message, generating the t Frame vision camera tracking target state vector collection Or the state vector set of the millimeter wave radar tracking target , update t Frame vision camera tracking target latest state vector queue collection , update t Frame millimeter wave radar tracking target latest state vector queue collection .
[0149] The present invention starts from step S2 and performs a fusion algorithm cycle. t Frame sensor message, generating the t Frame vision camera tracking target state vector collection Or the state vector set of the millimeter wave radar tracking target .
[0150] According to an embodiment of the present invention, if tThe frame sensor message is a visual camera tracking target message:
[0151] Generate t Frame vision camera tracking target state vector collection ;
[0152] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. i A visual camera tracks the target state vector; Indicates t The number of objects tracked by the frame vision camera.
[0153] The state vector of the present invention is used for matching the visual camera-millimeter wave radar tracking target, and the elements of the state vector can be freely selected according to the actual situation.
[0154] In one embodiment, t Frame No. i The visual camera tracks the target state vector Expressed in the following way:
[0155] ;
[0156] in, Indicates t Frame No. i A visual camera tracks the longitudinal coordinate of the target in the vehicle coordinate system; Indicates t Frame No. i A visual camera tracks the lateral coordinates of the target in the vehicle coordinate system; Indicates t Frame No. i A visual camera tracks the longitudinal velocity of the target in the vehicle coordinate system; Indicates t Frame No. i A visual camera tracks the lateral velocity of the target in the vehicle coordinate system; Indicates t Frame No. i The semantic type of the target tracked by a visual camera.
[0157] Use t The state vector set of the frame vision camera tracking target Update t The latest state vector queue collection of the frame vision camera tracking target :
[0158] Initialize tFrame vision camera tracking target latest state vector queue collection = . That is, initialize the t Frame vision camera tracking target latest state vector queue collection Equal to t-1 Frame vision camera tracking target latest state vector queue collection .
[0159] in, , Indicates t Frame No. i A visual camera tracks the target ID; Indicates t The number of targets tracked by the frame vision camera;
[0160] in, ;
[0161] Indicates the length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i A visual camera tracks the target state vector.
[0162] Traverse t The state vector set of the frame vision camera tracking target All visual cameras in the track target. That is, traverse Each visual camera tracks the target , , Indicates t The number of objects tracked by the frame vision camera.
[0163] Jordi t Frame vision camera tracking target latest state vector queue collection There is t Frame No. i A visual camera tracks the target The corresponding latest state vector queue ;
[0164] Then the t Frame No. i A visual camera tracks the target The state vector , increase to t Frame No. i A visual camera tracks the target The corresponding length is T No.t Frame No. i A visual camera tracks the target Latest state vector queue , and delete the queue with length T No. t Frame No. i A visual camera tracks the target The latest state vector queue The head element of the team.
[0165] Jordi t Frame vision camera tracking target latest state vector queue collection There is no t Frame No. i A visual camera tracks the target The corresponding latest state vector queue ;
[0166] The new length is T No. t Frame No. i A visual camera tracks the target Latest state vector queue , and the newly created length is T No. t Frame No. i A visual camera tracks the target Latest state vector queue Join t The latest state vector queue collection of the frame vision camera tracking target middle.
[0167] in, ;
[0168] Indicates that the length of the new T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i A visual camera tracks the target state vector.
[0169] After the traversal is completed, t Frame vision camera tracking target latest state vector queue collection The latest state vector queue that has not been updated in Element removal.
[0170] Update the following method t Frame millimeter wave radar tracking target latest state vector queue collection :
[0171] Initialize t The latest state vector queue collection of the frame millimeter wave radar tracking target = . That is, initialize the t Frame millimeter wave radar tracking target latest state vector queue collection Equal to t-1 Frame millimeter wave radar tracking target latest state vector queue collection .
[0172] in, , Indicates t Frame No. i Millimeter wave radar tracking target ID; Indicates t The number of targets tracked by the frame millimeter wave radar;
[0173] in, ;
[0174] Indicates the length is T No. t Frame No. i A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicates t Frame No. i A millimeter-wave radar tracking target state vector.
[0175] The first t The latest state vector queue collection of the frame millimeter wave radar tracking target In the example, each length is T No. t Frame No. i Millimeter wave radar tracks the target Latest state vector queue The head element of the team is deleted;
[0176] At the same time, each length is T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The tail element of the queue is copied and added to the tail of the queue, or in each length T No. t Frame No. i The latest state vector queue of the millimeter wave radar tracking target The tail element of the queue is padded with a zero vector.
[0177] According to an embodiment of the present invention, if t The frame sensor message is a millimeter wave radar tracking target message:
[0178] Generate t Frame millimeter wave radar tracking target state vector set ;
[0179] in, Indicates t Frame No. i Millimeter wave radar tracking target ID; Indicates t Frame No. i A millimeter-wave radar tracking target state vector; Indicates t Frame millimeter wave radar tracking number of targets.
[0180] In one embodiment, t Frame No. i The state vector of the target tracked by the millimeter-wave radar Expressed in the following way: ;
[0182] in, Indicates t Frame No. i The longitudinal coordinate of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates t Frame No. i The lateral coordinates of the target tracked by a millimeter-wave radar in the vehicle coordinate system; Indicates t Frame No. i A millimeter-wave radar tracks the longitudinal velocity of the target in the vehicle coordinate system; Indicates t Frame No. i A millimeter-wave radar tracks the lateral velocity of the target in the vehicle coordinate system; Indicates t Frame No. i The longitudinal acceleration of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates t Frame No. i The lateral acceleration of a millimeter-wave radar tracking target in the vehicle coordinate system; Indicates t Frame No. i The type of lane line where the millimeter-wave radar tracks the target.
[0183] Use t Frame millimeter wave radar tracking target state vector set Update t Frame millimeter wave radar tracking target latest state vector queue collection :
[0184] Initialize t Frame millimeter wave radar tracking target latest state vector queue collection = . That is, initialize the t The latest state vector queue collection of the frame millimeter wave radar tracking target Equal to t-1 Frame millimeter wave radar tracking target latest state vector queue collection .
[0185] in, , Indicates t Frame No. i Millimeter wave radar tracking target ID; Indicates t Number of targets tracked by frame millimeter wave radar;
[0186] in, ;
[0187] Indicates the length is T No. t Frame No. i A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicates t Frame No. i A millimeter-wave radar tracking target state vector.
[0188] Traverse t Frame millimeter wave radar tracking target state vector set All the millimeter wave radar tracking targets in. That is, traverse Each millimeter wave radar tracks the target , , Indicates t Frame millimeter wave radar tracking number of targets.
[0189] Jordi t Frame millimeter wave radar tracking target latest state vector queue collection There is t Frame No. i Millimeter wave radar tracks the target The corresponding latest state vector queue ;
[0190] Then the t Frame No. i Millimeter wave radar tracks the target The state vector , increase to t Frame No. iMillimeter wave radar tracks the target The corresponding length is T No. t Frame No. i Millimeter wave radar tracks the target Latest state vector queue , and delete the queue with length T Frame 1 of i Millimeter wave radar tracks the target Latest state vector queue The head element of the team.
[0191] Jordi t Frame millimeter wave radar tracking target latest state vector queue collection There is no t Frame No. i Millimeter wave radar tracks the target The corresponding latest state vector queue ;
[0192] The new length is T No. t Frame No. i Millimeter wave radar tracks the target Latest state vector queue , and the newly created length is T No. t Frame No. i Millimeter wave radar tracks the target Latest state vector queue Join t The latest state vector queue collection of the frame millimeter wave radar tracking target middle.
[0193] in, ;
[0194] Indicates that the length of the new T No. t Frame No. i A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicates t Frame No. i A millimeter-wave radar tracking target state vector.
[0195] After the traversal is completed, t Frame millimeter wave radar tracking target latest state vector queue collection The latest state vector queue that has not been updated in Element removal.
[0196] Update the following method tFrame vision camera tracking target latest state vector queue collection :
[0197] Initialize t Frame vision camera tracking target latest state vector queue collection = . That is, initialize the t The latest state vector queue collection of the frame vision camera tracking target Equal to t-1 Frame vision camera tracking target latest state vector queue collection .
[0198] in, , Indicates t Frame No. i A visual camera tracks the target ID; Indicates t The number of targets tracked by the frame vision camera;
[0199] in, ;
[0200] Indicates the length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i A visual camera tracks the target state vector.
[0201] The first t Frame vision camera tracking target Latest state vector queue collection In the example, each length is T No. t Frame No. i A visual camera tracks the target Latest state vector queue The head element of the team is deleted;
[0202] At the same time, each length is T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The tail element of the queue is copied and added to the tail of the queue, or in each length T No. t Frame No. i A queue of the latest state vectors of the visual camera tracking target The tail element of the queue is padded with a zero vector.
[0203] Step S3: t Each visual camera tracks the target with the t Each millimeter-wave radar tracking target is paired in each frame, and a time-series visual camera-millimeter-wave radar fusion state vector is constructed for each target pair. , input visual camera-millimeter wave radar tracking target timing matching classification network, output the matching type and matching probability of each target pair, and generate the first t Frame vision camera-millimeter wave radar fusion tracking target matching pair set .
[0204] In step S2 of the present invention, according to t The source of the frame sensor message is different (from the visual camera or from the millimeter wave radar), complete the t Frame vision camera tracking target latest state vector queue collection Updates, and t Frame millimeter wave radar tracking target latest state vector queue collection Updates.
[0205] In step S3, t The latest state vector queue collection of the frame vision camera tracking target Each target in (each visual camera tracks the target , ), and t The latest state vector queue collection of the frame millimeter wave radar tracking target Each target in (each millimeter wave radar tracking target , ) Pair them up in pairs to build a set of visual camera-millimeter wave radar fusion tracking target pairs .
[0206] Specifically, the t Each visual camera tracks the target with the t Each millimeter-wave radar tracking target in the frame is paired with each other to construct the t Frame vision camera-millimeter wave radar fusion tracking target pair collection , expressed as follows:
[0207] ;
[0208] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. j Millimeter wave radar tracking target ID; Indicatest The number of targets tracked by the frame vision camera; Indicates t Frame millimeter wave radar tracking number of targets.
[0209] Traverse the constructed t Visual camera-millimeter wave radar fusion tracking target pair collection Each visual camera-millimeter wave radar fusion target pair in , the corresponding time-series visual camera-millimeter wave radar fusion state vector is constructed by the following method :
[0210] From t Frame vision camera tracking target latest state vector queue collection Extract the t Frame No. i A visual camera tracks the target Latest state vector queue ;
[0211] From t Frame millimeter wave radar tracking target latest state vector queue collection Extract the t Frame No. j Millimeter wave radar tracks the target Latest state vector queue ;
[0212] in, Indicates the extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target; Indicates t Frame No. i The latest state vector of the visual camera tracking target (the extracted t Frame No. i The latest state vector queue of the visual camera tracking target The t elements);
[0213] Indicates the extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target The -T +1+t elements; Indicates the extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target The -T+2+t elements;
[0214] Indicates the extracted t Frame No. j A queue of the latest state vectors of the millimeter-wave radar tracking target; Indicates t Frame No. j The latest state vector of the millimeter-wave radar tracking target (the extracted new t Frame No. j The latest state vector queue of the target tracked by the millimeter-wave radar The t elements);
[0215] Indicates the extracted t Frame No. j The latest state vector queue of the millimeter wave radar tracking target The -T+1+t elements; Indicates the extracted t Frame No. j The latest state vector queue of the millimeter wave radar tracking target The -T+2+t elements;
[0216] T Indicates the extracted t Frame No. i A queue of the latest state vectors of the visual camera tracking target Length / extracted t Frame No. j The latest state vector queue of the millimeter wave radar tracking target Length.
[0217] The extracted t Frame No. i A visual camera tracks the target Latest state vector queue Each element in t Frame No. j Millimeter wave radar tracks the target Latest state vector queue Each element in is normalized by mean-variance and then concatenated to obtain the constructed time-series visual camera-millimeter wave radar fusion state vector:
[0218] ;
[0219] in, and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ;
[0220] Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. j Millimeter-wave radar tracking target ID.
[0221] The constructed t Visual camera-millimeter wave radar fusion tracking target pair collection Each visual camera-millimeter wave radar fusion target pair in Corresponding time-series visual camera-millimeter wave radar fusion state vector Input visual camera-millimeter wave radar time series tracking target matching classification network, output each target pair Match type and match probability , construct the t Frame vision camera-millimeter wave radar fusion target pair matching result set ;
[0222] in, For visual camera-millimeter wave radar fusion target pair The match type, Indicates mismatch. Indicates matching, represents the matching probability;
[0223] Traverse t Frame vision camera-millimeter wave radar fusion target pair matching result set Each target pair The matching results are:
[0224] For t Frame Each visual camera tracks the target and selects the matching type , matching probability Largest millimeter wave radar tracking target As a millimeter wave radar to match and track targets;
[0225] If the matching probability If it is less than the preset threshold, t Frame No. iA visual camera tracks the target and matches it with a millimeter-wave radar. preferably, the preset threshold is set to 0.7.
[0226] Build t Frame vision camera-millimeter wave radar fusion matching tracking target matching pair set Expressed in the following way:
[0227] ;
[0228] in, Indicates t Frame No. i A visual camera tracks the target ID; Indicates t Frame No. i The ID of the millimeter-wave radar tracking target matched with the visual camera tracking target; Indicates t The number of objects tracked by the frame vision camera.
[0229] The visual camera-millimeter wave radar time-series tracking target matching classification network used in the present invention is a pre-trained deep learning neural network.
[0230] The visual camera-millimeter wave radar time series tracking target matching classification network is trained using manually annotated data. The dataset contains multiple visual camera-millimeter wave radar fusion target pairs. Corresponding time-series visual camera-millimeter wave radar fusion state vector Fusion of multiple visual cameras and millimeter wave radars for target alignment The match type (annotation label).
[0231] Deep learning neural networks can select different types of deep learning neural networks according to actual needs, including but not limited to multi-layer perceptron (MLP), long short-term memory network (LSTM), recurrent neural network (RNN), etc., and can also adjust or increase the internal modules of the above basic models as needed to improve network performance, including but not limited to increasing data enhancement, improving feature encoding, increasing model parameter depth, etc.
[0232] The cost function for training the camera-millimeter-wave radar time-series tracking target matching classification network can be selected according to needs, including but not limited to the cross-entropy cost function (Cross-Entropy), focal loss (Focal-Loss), etc.
[0233] The visual camera-millimeter wave radar temporal tracking target matching classification network training is trained using the gradient descent method or its variants, including but not limited to the adaptive motion estimation algorithm (Adam), the root mean square propagation algorithm (RMSprop), etc.
[0234] The input of the visual camera-millimeter wave radar temporal tracking target matching classification network is for each visual camera-millimeter wave radar fusion target pair Corresponding time-series visual camera-millimeter wave radar fusion state vector , the output is for each visual camera-millimeter wave radar fusion target pair Match type and match probability .
[0235] Step S4: Update t Frame vision camera-millimeter wave radar fusion tracking target collection .
[0236] According to an embodiment of the present invention, the following method is used to update the t Frame vision camera-millimeter wave radar fusion tracking target collection :
[0237] Initialize t Frame vision camera-millimeter wave radar fusion tracking target collection = . That is, initialize the t A collection of frame vision camera-millimeter wave radar fusion tracking targets Equal to t-1 A collection of frame vision camera-millimeter wave radar fusion tracking targets .
[0238] Among them, t A collection of frame vision camera-millimeter wave radar fusion tracking targets Includes multiple t Frame vision camera-millimeter wave radar fusion tracking target , ;
[0239] No. t-1 A collection of frame vision camera-millimeter wave radar fusion tracking targets Includes multiple t-1 Frame vision camera-millimeter wave radar fusion tracking target , ;
[0240] in, Indicates t Frame No. k A visual camera-millimeter wave radar fusion tracks the target; Indicates t Frame No. i A visual camera tracks the target; Indicates t Frame No. iThe ID of the millimeter-wave radar tracking target matched with the visual camera tracking target; Indicates t The state vector of the frame vision camera-millimeter wave radar fusion tracking target;
[0241] Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target ; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracks the target; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding millimeter-wave radar tracks the target; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The state vector of .
[0242] Traversing the set of target matching pairs for visual camera-millimeter wave radar fusion tracking Each visual camera-millimeter wave radar fusion tracking target matching pair .
[0243] When t A collection of frame vision camera-millimeter wave radar fusion tracking targets In t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracking target satisfies At that time, t A collection of frame vision camera-millimeter wave radar fusion tracking targets Existence of hour;
[0244] Jordi t If the frame sensor message is a visual camera tracking target message, the t Frame No. i The visual camera tracks the target state vector As the t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0245] Jordi t If the frame sensor message is a millimeter wave radar tracking target message, the t Frame No. i The state vector of the target tracked by the millimeter-wave radar As the t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0246] in, ;
[0247] Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target ; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracking target ID; Indicates t- 1 Frame No. k A visual camera-millimeter wave radar fusion tracking target Corresponding millimeter-wave radar tracking target ID; Indicates t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The state vector of
[0248] The first t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector and t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The state vector , and input them into the Kalman filter together, so that t-1 State vector of target tracked by frame vision camera-millimeter wave radar fusion Update to t State vector of target tracked by frame vision camera-millimeter wave radar fusion .
[0249] At the same time, the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding millimeter wave radar tracks the target Updated to t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. , generating t Frame No. k A visual camera-millimeter wave radar fusion tracking target ;
[0250] in, ;
[0251] Indicates t Frame No. k A visual camera-millimeter wave radar fusion tracks the target; Indicates t Frame No. i A visual camera tracks the target; Indicates t Frame No. i The ID of the millimeter-wave radar tracking target matched with the visual camera tracking target; Indicates t The state vector of the frame vision camera-millimeter wave radar fusion tracking target;
[0252] The first t A collection of frame vision camera-millimeter wave radar fusion tracking targets Unupdated visual camera-millimeter wave radar fusion tracking target Delete, complete t A collection of frame vision camera-millimeter wave radar fusion tracking targets Updates.
[0253] When t A collection of frame vision camera-millimeter wave radar fusion tracking targets There is no t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracks the target satisfy At that time, t A collection of frame vision camera-millimeter wave radar fusion tracking targets Does not exist of hour;
[0254] Jordi t The frame sensor message is a visual camera tracking target message, and it will be t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. The corresponding state vector As the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0255] The first t Frame No. i The visual camera tracks the target state vector As the t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0256] Jordi t The frame sensor is a millimeter wave radar tracking target message, then the first t Frame No. i The visual camera tracks the target state vector As the t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector ;
[0257] Will be with t Frame No. i A visual camera tracks the target and matches it with a millimeter-wave radar. The corresponding state vector As the t Frame vision camera-millimeter wave radar fusion tracking target The observation vector ;
[0258] The first t-1 Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector and t Frame No. k A visual camera-millimeter wave radar fusion tracking target The observation vector , sequentially input into the Kalman filter, update the t Frame No. k The state vector of the target tracked by the visual camera-millimeter wave radar fusion ;
[0259] Initialize t Frame No. k A visual camera-millimeter wave radar fusion tracking target , will t Frame No.k A visual camera-millimeter wave radar fusion tracking target Join t A collection of frame vision camera-millimeter wave radar fusion tracking targets and t A collection of frame vision camera-millimeter wave radar fusion tracking targets Unupdated visual camera-millimeter wave radar fusion tracking target Delete, complete t A collection of frame vision camera-millimeter wave radar fusion tracking targets Updates.
[0260] Step S5: Get t+1 Frame sensor message, repeat steps S2 to S4.
[0261] When t If the frame sensor message is not the last frame, get the first t+1 Frame sensor message, repeat steps S2 to S4.
[0262] When t If the frame sensor message is the last frame, the fusion algorithm loop ends.
[0263] As shown in Table 1, a precision / recall / F1-score comparison table of a fusion method for tracking targets using a visual camera and a millimeter-wave radar in accordance with an embodiment of the present invention is provided.
[0264] Table 1 Comparison table of precision / recall / F1-score of the present invention
[0265] category Accuracy Recall F1-score Positive 95.80% 99.12% 97.44% Negative 99.23% 96.28% 97.73%
[0266] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A method for integrating visual camera and millimeter wave radar to track a target, characterized in that: The fusion method comprises the following steps: S1, obtain the first frame sensor message, generate the first frame visual camera tracking target state vector set α1 or the millimeter wave radar tracking target state vector set β1, initialize the first frame visual camera tracking target latest state vector queue set ψ1, initialize the first frame millimeter wave radar tracking target latest state vector queue set ζ1, initialize the first frame visual camera-millimeter wave radar fusion tracking target set Γ1; S2. Get the sensor message of the tth frame and generate the state vector set α of the visual camera tracking target of the tth frame t Or the state vector set β of the target tracked by the millimeter-wave radar t , update the latest state vector queue set ψ of the visual camera tracking target in the tth frame t , update the latest state vector queue set ζ of the millimeter wave radar tracking target in the tth frame t ; S3, pair each visual camera tracking target in the tth frame with each millimeter wave radar tracking target in the tth frame, and construct a time-series visual camera-millimeter wave radar fusion state vector for each target pair. Input the visual camera-millimeter wave radar tracking target temporal matching classification network, output the matching type and matching probability of each target pair, and generate the set Ω of the t-th frame visual camera-millimeter wave radar fusion tracking target matching pairs. t ; Among them, each visual camera tracking target in the tth frame is paired with each millimeter-wave radar tracking target in the tth frame to construct the tth frame visual camera-millimeter-wave radar fusion tracking target pair set A t , expressed as follows: in, Indicates the ID of the target tracked by the i-th visual camera in the t-th frame; Indicates the number of targets tracked by the visual camera in frame t; Indicates the ID of the j-th millimeter-wave radar tracking target in the t-th frame; Indicates the number of targets tracked by the millimeter-wave radar in the tth frame; Traverse the constructed t-th visual camera-millimeter wave radar fusion tracking target pair set A t Each visual camera-millimeter wave radar fusion target pair in The corresponding time-series visual camera-millimeter-wave radar fusion state vector is constructed by the following method The set of the latest state vector queue ψ of the target tracked by the visual camera in the tth frame t Extract the latest state vector queue of the i-th visual camera tracking target in the t-th frame From the latest state vector queue set ζ of the millimeter-wave radar tracking target in the tth frame t Extract the latest state vector queue of the jth millimeter-wave radar tracking target in the tth frame in, Represents the latest state vector queue of the i-th visual camera tracking target extracted in the t-th frame; Represents the latest state vector of the target tracked by the i-th visual camera in the t-th frame; Represents the latest state vector queue of the j-th millimeter-wave radar tracking target extracted in the t-th frame; Represents the latest state vector of the j-th millimeter-wave radar tracking target in the t-th frame; T represents the length of the latest state vector queue of the i-th visual camera tracking target extracted in the t-th frame / the length of the latest state vector queue of the j-th millimeter-wave radar tracking target extracted in the t-th frame; Queue the latest state vector of the i-th visual camera tracking target extracted in the t-th frame Each element in is the latest state vector queue of the j-th millimeter-wave radar tracking target extracted from the t-th frame Each element in is normalized by mean-variance and then concatenated to obtain the constructed time-series visual camera-millimeter wave radar fusion state vector: in, and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ; and for and The mean-variance normalization result of ; Each visual camera-millimeter-wave radar fusion target pair in the constructed t-th visual camera-millimeter-wave radar fusion tracking target pair set At Corresponding time-series visual camera-millimeter wave radar fusion state vector Input visual camera-millimeter wave radar time series tracking target matching classification network, output each target pair Match type and match probability Construct the matching result set of the t-th frame visual camera-millimeter wave radar fusion target pair in, For visual camera-millimeter wave radar fusion target pair The match type, Indicates mismatch. Indicates matching, represents the matching probability; Traverse each target pair in the t-th frame visual camera-millimeter wave radar fusion target pair matching result set P Matching results For each visual camera tracking target in frame t, select the matching type Matching probability Largest millimeter wave radar tracking target As a millimeter wave radar to match and track targets; If the matching probability If the value is less than the preset threshold, the millimeter-wave radar tracking target that matches the i-th visual camera tracking target in the t-th frame will be Set to 0; Construct the target matching pair set Ω of the t-th frame visual camera-millimeter wave radar fusion matching tracking t Expressed in the following way: S4, update the t-th frame visual camera-millimeter wave radar fusion tracking target set Γ t ; S5. Obtain the sensor message of the t+1th frame, and repeat steps S2 to S4.
2. The fusion method according to claim 1, characterized in that: In step S1, If the first frame sensor message is a visual camera tracking target message: Generate the first frame visual camera tracking target state vector set in, Indicates the ID of the target tracked by the i-th visual camera in the first frame; Represents the state vector of the target tracked by the i-th visual camera in the first frame; Indicates the number of targets tracked by the visual camera in the first frame; Initialize the latest state vector queue set of the visual camera tracking target in the first frame At the same time, initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target to an empty set And initialize the first frame of visual camera-millimeter wave radar fusion tracking target set in, Represents the latest state vector queue of the i-th visual camera tracking target in the first frame with a length of T; If the first frame of sensor message is a millimeter-wave radar tracking target message: Generate the first frame of millimeter wave radar tracking target state vector set in, Indicates the ID of the i-th millimeter-wave radar tracking target in the first frame; represents the state vector of the i-th millimeter-wave radar tracking target in the first frame; Indicates the number of targets tracked by the millimeter-wave radar in the first frame; Initialize the latest state vector queue set of the first frame of millimeter wave radar tracking target At the same time, initialize the latest state vector queue set of the visual camera tracking target in the first frame to an empty set And initialize the first frame visual camera-millimeter wave radar fusion tracking target set to an empty set in, Represents the latest state vector queue of the i-th millimeter-wave radar tracking target in the 1st frame with a length of T.
3. The fusion method according to claim 1, characterized in that: In step S2, If the sensor message of the tth frame is a visual camera tracking target message: Generate the state vector set of the visual camera tracking target at frame t Use the t-th frame visual camera to track the target state vector set α t Update the latest state vector queue set ψ of the visual camera tracking target in the tth frame t : Initialize the latest state vector queue set ψ of the visual camera tracking target in the tth frame t =ψ t-1 ; Traverse all visual cameras tracking targets in frame t; If the latest state vector queue set ψ of the visual camera tracking target in frame t t There is a queue of the latest state vectors corresponding to the target tracked by the i-th visual camera in the t-th frame; Then the target state vector of the i-th visual camera tracking target in the t-th frame is Add to the latest state vector queue of the i-th visual camera tracking target in the t-th frame with a length of T corresponding to the i-th visual camera tracking target in the t-th frame The tail of the queue and delete the latest state vector queue of the i-th visual camera tracking target in the t-th frame with a length of T The head element of the team; If the latest state vector queue set ψ of the visual camera tracking target in frame t t There is no latest state vector queue corresponding to the target tracked by the i-th visual camera in the t-th frame; Then a new queue of the latest state vector of the target tracked by the ith visual camera in the tth frame with a length of T is created. And create a new queue of length T for the latest state vector of the i-th visual camera tracking target in the t-th frame Join the latest state vector queue set ψ of the visual camera tracking target in the tth frame t middle; in, Indicates the newly created queue of the latest state vector of the target tracked by the ith visual camera in the tth frame with a length of T; After the traversal is completed, the latest state vector queue set ψ of the visual camera tracking target in the tth frame is t The latest state vector queue element that is not updated in is deleted; Update the latest state vector queue set ζ of the millimeter-wave radar tracking target in the tth frame by the following method t : Initialize the latest state vector queue set ζ of the target tracked by the millimeter-wave radar in the tth frame t =ζ t-1 ; The latest state vector queue set ζ of the millimeter-wave radar tracking target in the tth frame t In each of the t-th frame, the i-th millimeter-wave radar tracking target latest state vector queue is T. The head element of the team is deleted; At the same time, each length T of the latest state vector queue of the ith millimeter-wave radar tracking target in the tth frame The tail element of the queue is copied and added to the tail of the queue, or in each t-th frame of length T, the latest state vector queue of the i-th millimeter-wave radar tracking target The tail element of the queue is padded with a zero vector.
4. The fusion method according to claim 1, characterized in that: In step S2, If the sensor message in the tth frame is a millimeter-wave radar tracking target message: Generate the state vector set of the millimeter wave radar tracking target in the tth frame Use the t-th frame millimeter-wave radar to track the target state vector set β t Update the latest state vector queue set ζ of the millimeter wave radar tracking target in the tth frame t : Initialize the latest state vector queue set ζ of the target tracking target of the millimeter wave radar in the tth frame t =ζ t-1 ; Traverse all millimeter-wave radar tracking targets in frame t; If the latest state vector queue set ζ of the target tracked by the millimeter-wave radar in the tth frame t There is a queue of the latest state vectors corresponding to the i-th millimeter-wave radar tracking target in the t-th frame; Then the state vector of the target tracked by the i-th millimeter-wave radar in the t-th frame is The latest state vector queue of the i-th millimeter-wave radar tracking target in the t-th frame with a length of T is added to the i-th millimeter-wave radar tracking target in the t-th frame The tail of the queue is removed, and the latest state vector queue of the i-th millimeter-wave radar tracking target in the t-th frame with a length of T is deleted. The head element of the team; If the latest state vector queue set ζ of the target tracked by the millimeter-wave radar in the tth frame t There is no latest state vector queue corresponding to the i-th millimeter-wave radar tracking target in the t-th frame; Then a new queue of the latest state vector of the ith millimeter-wave radar tracking target in the tth frame with a length of T is created. And the newly created length T of the latest state vector queue of the ith millimeter-wave radar tracking target in the tth frame Join the latest state vector queue set ζ of the target tracked by the millimeter-wave radar in the tth frame t middle; in, Indicates the newly created queue of the latest state vector of the i-th millimeter-wave radar tracking target in the t-th frame with a length of T; After the traversal is completed, the latest state vector queue set ζ of the millimeter-wave radar tracking target in the tth frame is t The latest state vector queue element that is not updated in is deleted; Update the latest state vector queue set ψ of the visual camera tracking target in the tth frame by the following method t : Initialize the latest state vector queue set ψ of the visual camera tracking target in the tth frame t =ψ t-1 ; The latest state vector queue set ψ of the visual camera tracking target in the tth frame t In each t-th frame, the i-th visual camera tracking target latest state vector queue with length T The head element of the team is deleted; At the same time, each t-th frame i-th visual camera tracking target latest state vector queue with length T The tail element of the queue is copied and added to the tail of the queue, or in each t-th frame of length T, the i-th visual camera tracks the latest state vector queue of the target The tail element of the queue is padded with a zero vector.
5. The fusion method according to claim 1, characterized in that: In step S3, the visual camera-millimeter wave radar temporal tracking target matching classification network used is a pre-trained deep learning neural network; The visual camera-millimeter wave radar time series tracking target matching classification network is trained using manually annotated data. The dataset contains multiple visual camera-millimeter wave radar fusion target pairs. Corresponding time-series visual camera-millimeter wave radar fusion state vector Fusion of multiple visual cameras and millimeter wave radars for target alignment Match type 6. The fusion method according to claim 1, characterized in that: In step S4, the t-th frame visual camera-millimeter wave radar fusion tracking target set Γ is updated by the following method: t : Initialize the t-th frame visual camera-millimeter wave radar fusion tracking target set Γ t =Γ t-1 ; Traversing the set of matching pairs of visual camera-millimeter wave radar fusion tracking targets Ω t Each visual camera-millimeter wave radar fusion tracking target matching pair in When the visual camera-millimeter wave radar fusion tracking target set Γ in the tth frame t In the t-1th frame, there is a kth visual camera-millimeter wave radar fusion tracking target The corresponding visual camera tracks the target satisfy hour; If the sensor message of the t-th frame is a visual camera tracking target message, then the state vector of the i-th visual camera tracking target of the t-th frame is As the observation vector of the kth visual camera-millimeter wave radar fusion tracking target in the tth frame If the sensor message of the t-th frame is a millimeter-wave radar tracking target message, then the state vector of the i-th millimeter-wave radar tracking target in the t-th frame is As the observation vector of the kth visual camera-millimeter wave radar fusion tracking target in the tth frame in, Indicates the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame Indicates the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame The corresponding visual camera tracking target ID; Indicates the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame The corresponding millimeter-wave radar tracks the target; Indicates the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame The state vector of The observation vector of the k-th visual camera-millimeter wave radar fusion tracking target in the t-th frame is Fusion tracking target with the kth visual camera-millimeter wave radar in frame t-1 The state vector Input them together into the Kalman filter to make the state vector of the t-1 frame visual camera-millimeter wave radar fusion tracking target Update to the state vector of the target tracked by the visual camera-millimeter wave radar fusion at frame t At the same time, the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame Corresponding millimeter wave radar tracking target Updated to the millimeter-wave radar tracking target that matches the i-th visual camera tracking target in the t-th frame Generate the kth visual camera-millimeter wave radar fusion tracking target in the tth frame in, represents the kth visual camera-millimeter wave radar fusion tracking target in the tth frame; represents the ID of the millimeter-wave radar tracking target that matches the i-th visual camera tracking target in the t-th frame; Represents the state vector of the target tracked by visual camera-millimeter-wave radar fusion at frame t; The visual camera-millimeter wave radar fusion tracking target set Γt in the t-th frame is updated by the visual camera-millimeter wave radar fusion tracking target. Delete and complete the update of the set Γt of the t-th frame visual camera-millimeter wave radar fusion tracking targets.
7. The fusion method according to claim 6, characterized in that: When the set Γt of the visual camera-millimeter wave radar fusion tracking target in the t-th frame does not exist in the k-th visual camera-millimeter wave radar fusion tracking target in the t-1th frame The corresponding visual camera tracks the target satisfy hour; If the sensor message of the t-th frame is a visual camera tracking target message, then it will be matched with the i-th visual camera tracking target of the t-th frame. Matching millimeter wave radar tracking target The corresponding state vector As the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame The observation vector The i-th visual camera tracking target state vector of the t-th frame As the kth visual camera-millimeter wave radar fusion tracking target in the tth frame The observation vector If the sensor of the t-th frame is a millimeter-wave radar tracking target message, then the i-th visual camera tracking target state vector of the t-th frame is As the kth visual camera-millimeter wave radar fusion tracking target in the t-1th frame The observation vector The target will be tracked with the i-th visual camera in the t-th frame Matching millimeter wave radar tracking target The corresponding state vector As the t-th frame visual camera-millimeter wave radar fusion tracking target The observation vector Df t k ; The kth visual camera and millimeter wave radar are fused to track the target in the t-1th frame The observation vector And the kth visual camera-millimeter wave radar fusion tracking target in the tth frame The observation vector Df t k , sequentially input into the Kalman filter, update the kth visual camera-millimeter wave radar fusion tracking target in the tth frame The state vector Initialize the kth visual camera-millimeter wave radar fusion tracking target at the tth frame The kth visual camera-millimeter wave radar fusion tracking target in the tth frame Add to the set Γ of visual camera-millimeter wave radar fusion tracking targets in the tth frame t In the t-th frame, the visual camera-millimeter wave radar fusion tracking target set Γ t Unupdated visual camera-millimeter wave radar fusion tracking target Delete, and complete the set Γ of the t-th frame visual camera-millimeter wave radar fusion tracking target t Updates.
Citation Information
Patent Citations
Regional target tracking method fusing millimeter wave radar and camera
CN115184917A
Road monitoring fusion tracking and positioning speed measurement method based on camera and millimeter wave radar
CN115436927A
Matching fusion obstacle detection method and system, electronic equipment and storage medium
CN115909281A
Obstacle detection method based on camera and millimeter wave radar data fusion
CN116413725A
Target tracking method and system based on radar data and video data fusion
CN117949942A
