Multi-target tracking method, device, storage medium and product based on trajectory fusion

Abstract

The present invention discloses a multi-target tracking method, device and storage medium based on trajectory fusion, the method comprising detecting and tracking a video stream collected by a visual sensor to obtain a video target trajectory and a video target tracking number; processing and tracking a target echo signal detected by a radar to obtain a radar target trajectory and a radar target tracking number; aligning the video target trajectory and the radar target trajectory according to the frame rate of the video stream; associating and fusing the video target trajectory and the radar target trajectory after the frame rate alignment to obtain a radar target trajectory; generating a tracking number according to the type of the target trajectory and the information of the target trajectory. The present invention ensures the uniqueness of the target tracking number, improves the accuracy of multi-target tracking, and solves the problem of multi-target and tracking number switching during the fusion process of radar targets and video targets.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a multi-target tracking method, device, storage medium and product based on trajectory fusion. Background Art

[0002] With the improvement of social development, the number of motor vehicles on the road has increased, and incidents such as congestion, traffic accidents, and illegal lane changes have also increased, which has put forward higher requirements for traffic perception data. Target tracking technology integrating multiple sensors has become a necessary condition for the digital and intelligent transformation of road traffic control. Urban road management urgently needs to accurately obtain the target information and traffic statistics of the current road to provide data support for algorithms such as timing.

[0003] Usually, the video target detection algorithm is used to process the video stream captured by the camera, which can detect the appearance information, semantic information and number of targets in real time, but the target speed and target position cannot be obtained. In addition, the effect of video detection will be greatly reduced in bad weather such as night with insufficient light, foggy days, heavy rain, etc. Millimeter-wave radar measures the distance and speed of moving targets through the Doppler effect and is not affected by light. However, due to the insufficient amount of information obtained by millimeter-wave radar, it cannot accurately classify and identify targets, and it is easy to miss detection in scenes with a large number of targets and slow targets. In order to achieve accurate analysis of road targets in multiple scenarios and all-weather conditions, it is necessary to fuse radar targets and video targets before tracking the targets.

[0004] Existing target trackers associate and filter target information output by a sensor. Due to the inherent defects of sensors, the adaptability and performance of target tracking are limited.

[0005] The radar target tracker processes the received radar echo signal through radar signal processing technology, and then uses clustering algorithm, data association algorithm and filtering algorithm to filter and associate the processed signal, and finally outputs the state of the radar target. Millimeter wave radar detects the position and speed of moving targets through the Doppler effect, and cannot detect stationary targets. When there are many targets and the targets are moving slowly, the target detection ability of the millimeter wave radar will deteriorate, which will lead to a worse radar target tracking effect.

[0006] The video target tracker processes the video frames through video detection technology to obtain the pixel position and category of the target, and then filters and associates the position of the target in the previous and next video frames. When the target is far away from the camera, the pixels occupied by the target in the image are reduced, which makes it easy for the video detection algorithm to miss detection; when the external environment is in a scene with low visibility such as night, rainy days and haze, the detection effect of the video detection algorithm will also be reduced, which will lead to poor video target detection effect.

[0007] In order to improve the target tracking effect, the radar target information and the video target information are data-associated according to the transformation matrix, and then a target tracker is established based on the target that integrates the radar information and visual information to achieve target tracking. Since the target detection part of the millimeter-wave radar is unstable, the received target position will shake, causing the fusion of the radar target and the video target to fail, causing the target to be misidentified as two targets, resulting in the phenomenon of multiple targets during target tracking. When the target is far away from the camera, the slight jitter of the video target can easily increase the calibration error of the radar target and the video target, resulting in the phenomenon of switching the target tracking sequence number.

[0008] For example, the patent document with the announcement number CN114299417A and the name of a multi-target tracking method based on radar-visual fusion discloses the use of perspective transformation to associate radar targets and video targets, and to fuse the radar-visual information and store it in the tracker; predict the next state of the target in the tracker, and associate the predicted target and observed target data information and perform trajectory matching; then use the filtering algorithm to update the associated target to complete multi-target tracking. This method integrates radar and video information, improves the problem of unstable target tracking caused by inaccurate target detection caused by single sensor detection information, but cannot solve the problem of unstable calibration errors of radar targets and video targets, which easily leads to the problem of switching between multiple targets and tracking numbers after multi-target tracking.

[0009] In summary, the existing multi-target tracking algorithms have poor multi-target tracking capabilities due to the inherent detection defects of different sensors; at the same time, the multi-target tracking algorithms based on the correlation between radar targets and video targets are prone to multi-target and tracking sequence number switching problems after multi-target tracking due to the unstable calibration errors of radar targets and video targets. Summary of the invention

[0010] The purpose of the present invention is to provide a multi-target tracking method, device, storage medium and product based on trajectory fusion, so as to solve the problem that the multi-target tracking technology based on radar-vision fusion easily leads to the switching of multiple targets and tracking numbers after multi-target tracking.

[0011] The present invention solves the above technical problems through the following technical solutions: a multi-target tracking method based on trajectory fusion, comprising the following steps:

[0012] Detect and track the video stream collected by the visual sensor to obtain the video target trajectory and video target tracking number; process and track the target echo signal detected by the radar to obtain the radar target trajectory and radar target tracking number;

[0013] Performing frame rate alignment on the video target trajectory and the radar target trajectory according to the frame rate of the video stream;

[0014] The video target trajectory and radar target trajectory after frame rate alignment are associated and fused to obtain the radar target trajectory;

[0015] When the target trajectory is a video target trajectory, a tracking sequence number is generated according to the target confidence and duration of the video target trajectory;

[0016] When the target track is a radar target track, a tracking sequence number is generated according to the target speed and duration of the radar target track;

[0017] When the target trajectory is a radar-sighted target trajectory, a tracking number is generated according to the target confidence and duration of the radar-sighted target trajectory, or a tracking number is generated according to the target speed and duration of the radar-sighted target trajectory.

[0018] Further, according to the video target information obtained by detecting the video stream, a video target tracker is used to track the video target to obtain a video target trajectory; according to the radar target information obtained by processing the target echo signal, a radar target tracker is used to track the radar target to obtain a radar target trajectory; the specific process of the video target tracker or the radar target tracker performing target tracking includes:

[0019] Cache the previous frame of target information, wherein the target information is video target information or radar target information;

[0020] The target information of the current frame is used as the observation value, and the filtering algorithm is used to predict the target information of the previous frame to obtain the predicted value;

[0021] Calculating a similarity matrix between the observed values ​​and the predicted values;

[0022] The optimal matching matrix of the similarity matrix is ​​calculated by using the KM matching algorithm, and the optimal similarity between each observed value and the corresponding predicted value is determined according to the optimal matching matrix of the similarity matrix;

[0023] When the optimal similarity exceeds the set similarity threshold, the observed value and the corresponding predicted value are the same target, and the target is successfully associated in two adjacent frames; when the optimal similarity is less than the set similarity threshold, the observed value is a new target appearing in the current frame, and the new target information is cached;

[0024] When the target is successfully associated between consecutive frames, a target tracking number is assigned; when the target is failed to be associated between consecutive frames, the target information in the target tracker is deleted, wherein the target tracking number is a video target tracking number or a radar target tracking number, and the target tracker is a video target tracker or a radar target tracker.

[0025] Further, the frame rate of the video target track and the radar target track are aligned according to the frame rate of the video stream, specifically including:

[0026] Obtain the timestamps of the video target trajectory and the radar target trajectory, wherein the timestamp of each video target in the video target trajectory is the moment when the target is detected on the corresponding frame image of the video stream, and the timestamp of each radar target in the radar target trajectory is the moment when the radar receives the corresponding target echo signal;

[0027] Based on the frame rate of the video stream and the timestamp of each radar target in the radar target track, when the i-th frame of the radar target track has target information, the i-th frame of the radar target track is not interpolated; when the i-th frame of the radar target track does not have target information, the i-th frame of the radar target track is interpolated according to the target information of the i-1th frame of the radar target track.

[0028] Furthermore, the KM matching algorithm is used to associate and fuse the video target trajectory and radar target trajectory after frame rate alignment, including:

[0029] Calculate the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment;

[0030] The KM matching algorithm is used to calculate the optimal matching matrix of the cost matrix, and the video target trajectory and the radar target trajectory are associated and fused according to the optimal matching matrix of the cost matrix to obtain the radar target trajectory.

[0031] Furthermore, the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment is calculated, which specifically includes the following steps:

[0032] Each video target track is converted into a video mapping target track in the radar coordinate system according to the conversion matrix between the radar coordinate system and the camera coordinate system;

[0033] Calculate the cost trajectory sequence between the video mapping target trajectory and the radar target trajectory and the standard deviation σ of the cost trajectory sequenceij and mean μ ij , the specific calculation formula is:

[0034] X ij =|V i -R j |

[0035] Among them, X ij represents the cost trajectory sequence between the i-th video mapping target trajectory and the j-th radar target trajectory, X ij ={x ij1 ,x ij2 ,…,x ijk ,…,x ijn}, x ijk Represents the cost trajectory sequence X ij The kth value of V i represents the target trajectory of the i-th video mapping, V i = {v i1 ,v i2 ,…,v ik ,…,v in},v ik Represents the coordinate information of the video target at the kth moment in the i-th video mapping target trajectory; R j represents the jth radar target track, R j = {r j1 ,r j2 ,…,r jk ,…,r jn}, r jk represents the coordinate information of the radar target at the kth moment in the jth radar target trajectory; n represents the length of the cost trajectory sequence or the video mapping target trajectory or the radar target trajectory;

[0036] Take the cost trajectory sequence X ij In {σ ij -μ ij ,σ ij +μ ij} range as a valid trajectory sequence According to the cost trajectory sequence X ij And the effective trajectory sequence The cost value between the i-th video mapping target track and the j-th radar target track is calculated, and then the cost matrix between all video target tracks and all radar target tracks is obtained.

[0037] Furthermore, the specific calculation formula of the cost value between the i-th video mapping target track and the j-th radar target track is:

[0038]

[0039] Among them, C ij represents the cost between the i-th video mapping target track and the j-th radar target track, D t Indicates the set distance error threshold, P t represents the set probability threshold, Represents a valid trajectory sequence The average of the values ​​in .

[0040] Further, the generating of the tracking sequence number according to the target confidence and duration of the video target trajectory specifically includes:

[0041] Calculate the average value of the target confidence in the target trajectory of the video, and use the average value as the initial batch confidence of the current target;

[0042] The confidence threshold for the start batch is calculated according to the duration of the target trajectory of the video. The specific formula is:

[0043]

[0044] Among them, CDT represents the confidence threshold of the batch, CDT max Indicates the maximum confidence threshold for the batch, CDT min Indicates the minimum confidence threshold for the batch, Tv max Indicates the maximum batch duration of the set video target track, Tv min Indicates the minimum batch duration of the set video target trajectory, and Tv indicates the duration of the video target trajectory;

[0045] The tracking sequence number is generated according to the batch confidence of the current target, the batch confidence threshold CDT and the duration Tv of the video target trajectory.

[0046] Furthermore, a tracking sequence number is generated according to the batch confidence of the current target, the batch confidence threshold CDT and the duration Tv of the video target trajectory, specifically including:

[0047] When the batch confidence of the current target is greater than the batch confidence threshold CDT, a tracking number is generated;

[0048] When the batch confidence of the current target is less than or equal to the batch confidence threshold CDT, and Tv≤Tv max When , no tracking number is generated;

[0049] When the batch confidence of the current target is less than or equal to the batch confidence threshold CDT, and Tv>Tv max , delete the video target track.

[0050] Further, the generating of the tracking sequence number according to the target speed and duration of the radar target track specifically includes:

[0051] Calculate the average value of the target speed in the radar target track, and use the average value as the starting speed of the current target;

[0052] The batch starting speed threshold is calculated according to the duration of the radar target track. The specific formula is:

[0053]

[0054] Among them, ST represents the batch speed threshold, ST max Indicates the maximum batch speed threshold, ST min Indicates the minimum batch speed threshold, Tr max Indicates the maximum batch duration of the radar target track, Tr min Indicates the minimum batch duration of the radar target track, Tr indicates the duration of the radar target track;

[0055] The tracking sequence number is generated according to the batch speed of the current target, the batch speed threshold ST and the duration Tr of the radar target track.

[0056] Furthermore, a tracking sequence number is generated according to the batch speed of the current target, the batch speed threshold ST and the duration Tr of the radar target track, specifically including:

[0057] When the batch start speed of the current target is greater than the batch start speed threshold ST, a tracking number is generated;

[0058] When the batch start speed of the current target is less than or equal to the batch start speed threshold ST, and Tr≤Tr max When , no tracking number is generated;

[0059] When the batch start speed of the current target is less than or equal to the batch start speed threshold ST, and Tr>Tr max , delete the radar target track.

[0060] Based on the same concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the multi-target tracking method as described above.

[0061] Based on the same concept, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the multi-target tracking method as described above is implemented.

[0062] Based on the same concept, the present invention also provides a computer program product, including a computer program / instruction, which implements the multi-target tracking method as described above when executed by a processor.

[0063] Beneficial Effects

[0064] Compared with the prior art, the advantages of the present invention are:

[0065] The present invention regenerates the tracking number according to the type of target trajectory, the confidence or speed and the duration of the target trajectory, thereby ensuring the uniqueness of the target tracking number, improving the accuracy of multi-target tracking, and solving the problem of multi-target and tracking number switching in the process of radar target and video target fusion.

[0066] The present invention converts the video target trajectory into the video mapping target trajectory in the radar coordinate system according to the conversion matrix between the radar coordinate system and the camera coordinate system, then calculates the cost matrix between the video mapping target trajectory and the radar target trajectory, adopts the KM matching algorithm to calculate the optimal matching matrix of the cost matrix, obtains the optimal matching between the radar target trajectory and the video target trajectory according to the optimal matching matrix, and associates and fuses the optimally matched video target trajectory and radar target trajectory, thereby improving the problem that the instability of the target part detected by the millimeter-wave radar and the increase of mapping error caused by the jitter of the long-distance video target frame, which further leads to the failure of matching the radar target and the video target, and improves the trajectory stability of the multi-target tracker. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 is a flow chart of a multi-target tracking method based on trajectory fusion in an embodiment of the present invention;

[0069] Figure 2 2 is a schematic diagram of a frame rate alignment operation in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following is a clear and complete description of the technical solution in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0072] Example 1

[0073] like Figure 1 As shown, a multi-target tracking method based on trajectory fusion provided by an embodiment of the present invention includes the following steps:

[0074] Step S1: Detect and track the video stream collected by the visual sensor to obtain the video target trajectory and the video target tracking number; process and track the target echo signal detected by the radar to obtain the radar target trajectory and the radar target tracking number;

[0075] Step S2: aligning the frame rates of the video target trajectory and the radar target trajectory according to the frame rate of the video stream;

[0076] Step S3: Associating and fusing the video target trajectory and the radar target trajectory after frame rate alignment to obtain the radar target trajectory;

[0077] Step S4: Generate a tracking number according to the type of the target trajectory and the information of the target trajectory.

[0078] The multi-target tracking method based on radar and visual sensors uses visual sensors to collect the target video stream in real time, calls the target detection model to detect the target for each frame of the video stream, and obtains the video target information. The target detection model can call the existing target detection model, or build a new sample data set and target detection model. The sample data set is used to train and verify the target detection model to obtain the trained target detection model to detect the target for each frame of the video stream; the video target information includes the target box, target category, target confidence, and the timestamp of the video frame. After target detection, the video target information of each frame of the video stream is obtained. The video target tracker tracks the video target according to the video target information of each frame of the video stream to obtain the video target trajectory and the video target tracking sequence number.

[0079] The target echo signal is detected in real time by using the millimeter wave radar, and the target echo signal is processed by the radar signal processing algorithm to obtain the radar target information, wherein the radar target information includes the target position, target speed and radar target timestamp. After being processed by the radar signal processing algorithm, the radar target information of each frame of the echo signal can be obtained. The radar target tracker tracks the radar target according to the radar target information of each frame of the echo signal to obtain the radar target trajectory and the radar target tracking sequence number.

[0080] In a specific embodiment of the present invention, the specific process of the video object tracker performing object tracking includes:

[0081] Step S1.11: The video target tracker caches the first frame of video target information;

[0082] Step S1.12: using the target information of the second frame of the video detected by the target as the observation value, using the filtering algorithm in the video target tracker to make a prediction based on the cached target information of the first frame of the video to obtain a predicted value;

[0083] Step S1.13: Calculate the similarity matrix between the observed value and the predicted value, where the similarity matrix measures the similarity between the observed value and the predicted value;

[0084] Step S1.14: The association between the previous and next frames is to calculate the optimal matching matrix of the similarity matrix through the KM matching algorithm (Kuhn-Munkres Algorithm), and find the optimal similarity between each observation value and the corresponding predicted value according to the optimal matching matrix of the similarity matrix. When the optimal similarity exceeds the set similarity threshold, the observation value and the corresponding predicted value are the same video target, and the video target is successfully associated in two adjacent frames; when the optimal similarity is less than the set similarity threshold, the observation value is a new video target that appears in the second frame, and the video target tracker caches the new video target information. When the video target is successfully associated between consecutive frames, the video target tracking number is assigned; when the video target fails to be associated between consecutive frames, it indicates that the video target disappears, and the video target information in the video target tracker is deleted.

[0085] Similarly, the specific process of target tracking by the radar target tracker includes:

[0086] Step S1.21: The radar target tracker caches the first frame of radar target information;

[0087] Step S1.22: using the second frame radar target information as the observation value, using the filtering algorithm in the radar target tracker to make a prediction based on the cached first frame radar target information to obtain a predicted value;

[0088] Step S1.23: Calculate the similarity matrix between the observed values ​​and the predicted values;

[0089] Step S1.24: The association between the previous and next frames is to calculate the optimal matching matrix of the similarity matrix through the KM matching algorithm, and find the optimal similarity between each observation value and the corresponding predicted value according to the optimal matching matrix of the similarity matrix. When the optimal similarity exceeds the set similarity threshold, the observation value and the corresponding predicted value are the same radar target, and the radar target is successfully associated in two adjacent frames; when the optimal similarity is less than the set similarity threshold, the observation value is a new radar target appearing in the second frame, and the radar target tracker caches the new radar target information. When the radar target is successfully associated between consecutive frames, the radar target tracking number is assigned; when the radar target fails to be associated between consecutive frames, it indicates that the radar target disappears, and the radar target information in the radar target tracker is deleted.

[0090] The video target tracking number and radar target tracking number are values ​​within a certain range, and the maximum value within the range is greater than the maximum number of targets detected in the current frame, ensuring that the target tracking numbers output in the current frame are different. For example, the video target tracking number is a value within 1 to 3500, and the radar target tracking number is a value within 3501 to 7000.

[0091] The video target trajectory and radar target trajectory are cached according to their respective target tracking numbers. The specific process is: cache the target information assigned with the tracking number in the previous frame, and add the target information to the target trajectory corresponding to the tracking number; when the next frame has the same tracking number as the previous frame, update the target information corresponding to the tracking number, and add the target information to the target trajectory corresponding to the tracking number; when the next frame does not have the same tracking number as the previous frame (that is, the next frame is a new tracking number), add the target information of the new tracking number in the next frame to the target trajectory corresponding to the new tracking number.

[0092] When the target information corresponding to the tracking number is not updated for multiple consecutive frames, the tracking number and the target track corresponding to the tracking number are deleted.

[0093] In order to ensure the consistency of the length of the video target trajectory and the radar target trajectory in the same time period, improve the matching accuracy between the video target trajectory and the radar target trajectory and the real-time performance of the target information, the frame rate of the video target trajectory and the radar target trajectory are aligned according to the frame rate of the video stream. The specific process is as follows:

[0094] Step S2.1: Obtain the timestamps of the video target trajectory and the radar target trajectory, wherein the timestamp of each video target in the video target trajectory is the moment when target detection is performed on the corresponding frame image of the video stream, that is, the timestamp of the video target is the moment when the video target detection is performed using the target detection model; the timestamp of each radar target in the radar target trajectory is the moment when the radar receives the corresponding target echo signal, that is, the timestamp of the radar target is the moment when the radar receives the echo signal of the radar target.

[0095] Step S2.2: Based on the frame rate of the video stream and the timestamp of each radar target in the radar target track, when the i-th frame of the radar target track has target information, the i-th frame of the radar target track is not interpolated; when the i-th frame of the radar target track does not have target information, the i-th frame of the radar target track is interpolated according to the target information of the i-1-th frame of the radar target track; when the radar target track has no target information for n consecutive frames, the radar target track is not interpolated, and the target disappears, such as Figure 2As shown in the figure, n is the maximum number of interpolated frames, which is related to the frame rates of the radar and visual sensors. For example, when the radar frame rate is 10 and the visual sensor frame rate is 25, the maximum number of interpolated frames n is 3.

[0096] Millimeter-wave radar and visual sensors each have their own advantages. For example, millimeter-wave radar has strong anti-interference capabilities, can detect the distance, speed and direction of moving targets, is less affected by light and weather factors, and has all-weather working capabilities. However, the resolution of millimeter-wave radar is relatively low and cannot identify the characteristic information of the target; visual sensors can detect the appearance information of the target in real time with high accuracy, but visual sensors are easily affected by factors such as light and weather and cannot accurately estimate the target distance. In order to improve the accuracy of target detection, it is necessary to associate and fuse the video target trajectory and radar target trajectory after frame rate alignment to obtain the radar target trajectory. In a specific embodiment of the present invention, the KM matching algorithm is used to associate and fuse the video target trajectory and radar target trajectory after frame rate alignment, specifically including:

[0097] Step S3.1: Calculate the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment;

[0098] Step S3.2: The KM matching algorithm is used to calculate the optimal matching matrix of the cost matrix, and the video target trajectory and the radar target trajectory are associated and fused according to the optimal matching matrix of the cost matrix to obtain the radar target trajectory.

[0099] In step S3.1, the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment is calculated, which specifically includes the following steps:

[0100] Step 3.11: Convert each video target track into a video mapping target track in the radar coordinate system according to the transformation matrix between the radar coordinate system and the camera coordinate system, where V i represents the target trajectory of the i-th video mapping, V i = {v i1 ,v i2 ,…,v ik ,…,v in},v ik Represents the coordinate information of the video target at the kth moment in the i-th video mapping target trajectory; R j represents the jth radar target track, R j = {r j1 ,r j2 ,…,r jk ,…,r jn}, r jkrepresents the coordinate information of the radar target at the kth moment in the jth radar target track; n represents the length of the video mapping target track or the radar target track. Since the video target track and the radar target track have achieved frame rate alignment, the length of the video mapping target track corresponding to the video target track is equal to the length of the radar target track, which is n.

[0101] Step 3.12: Calculate the cost trajectory sequence X between the video mapping target trajectory and the radar target trajectory ij And the standard deviation σ of the cost trajectory sequence ij and mean μ ij , the specific calculation formula is:

[0102] X ij =|V i -R j | (1)

[0103] Among them, X ij represents the cost trajectory sequence between the i-th video mapping target trajectory and the j-th radar target trajectory, X ij ={x ij1 ,x ij2 ,…,x ijk ,…,x ijn}, x ijk Represents the cost trajectory sequence X ij The kth value of .

[0104] Step 3.13: Replace the valence trajectory sequence X ij In {σ ij -μ ij ,σ ij +μ ij} range as a valid trajectory sequence According to the cost trajectory sequence X ij And the effective trajectory sequence Calculate the cost value between the i-th video mapping target track and the j-th radar target track, and then get the cost matrix between all video target tracks and all radar target tracks. The specific formula is:

[0105]

[0106] Among them, C ij represents the cost between the i-th video mapping target track and the j-th radar target track, D t Indicates the set distance error threshold, P t represents the set probability threshold, Represents a valid trajectory sequence The average value of each value in. Therefore, the cost matrix C is C = {C ij,i=1,2,…,N,j=1,2,…,M}, N represents the number of video mapping target trajectories, and M represents the number of radar target trajectories. ij The probability threshold is set according to the distribution of the values ​​in . The probability threshold P in this embodiment t Set to 0.6826; set according to the accuracy of the video target mapped to the radar coordinate system, the distance error threshold D of the cost trajectory sequence of this embodiment t Set to 2 to 4 meters.

[0107] In step S3.2, the KM matching algorithm is used to calculate the optimal matching matrix of the cost matrix, and the video target trajectory and the radar target trajectory are associated and fused according to the optimal matching matrix of the cost matrix as the prior art.

[0108] The video target trajectory and the radar target trajectory are associated and fused according to the optimal matching matrix of the cost matrix, that is, the video mapping target trajectory corresponding to each radar target trajectory is obtained according to the optimal matching matrix, the video target trajectory is associated with the radar target trajectory according to the video target tracking sequence number and the radar target tracking sequence number, and then the target information of the video target trajectory and the radar target trajectory are fused to obtain the radar target trajectory. The radar target trajectory obtained after the video target trajectory and the radar target trajectory are successfully associated and fused has the video target tracking sequence number and the radar target tracking sequence number, and the target information of each frame in the radar target trajectory includes the video target information and the radar target information. The video target trajectory and the radar target trajectory fail to be associated, and the video target trajectory and the radar target trajectory cannot be fused, so it is still a separate video target trajectory or radar target trajectory, corresponding to which there is a video target tracking sequence number or a radar target tracking sequence number.

[0109] In order to ensure the uniqueness and accuracy of the target tracking number and avoid the problem of switching between multiple targets and tracking numbers after multi-target tracking, the target tracking number is regenerated after the video target track and the radar target track are associated and fused, that is, the start batch of the target track. Since the target track association and fusion are divided into two cases, success or failure, the types of target tracks after association and fusion include video target tracks, radar target tracks and radar target tracks (radar target tracks are obtained after successful association and fusion). Different target track types use different tracking number generation methods, that is, different start batch processes.

[0110] When the target trajectory is a video target trajectory, a tracking number is generated according to the target confidence and duration of the video target trajectory. The specific process includes:

[0111] Step S4.11: Calculate the average value of the target confidence in the video target trajectory, that is, calculate the average value of the target confidence of each frame in the video target trajectory, and use the average value as the starting batch confidence of the current target;

[0112] Step S4.12: Calculate the batch confidence threshold according to the duration of the target trajectory in the video. The specific formula is:

[0113]

[0114] Among them, CDT represents the confidence threshold of the batch, CDT max Indicates the maximum confidence threshold for the batch, CDT min Indicates the minimum confidence threshold for the batch, Tv max Indicates the maximum batch duration of the set video target track, Tv min Indicates the minimum batch duration of the set video target track, and Tv indicates the duration of the video target track.

[0115] The longer the duration of the video target trajectory, the lower the confidence threshold for the first batch; conversely, the shorter the duration of the video target trajectory, the higher the confidence threshold for the first batch. The confidence threshold for the first batch is set to 0.5-0.9, that is, the maximum confidence threshold for the first batch CDT max Set to 0.9, the minimum batch confidence threshold CDT min The batch start time refers to the target track time required to regenerate the tracking number. In this embodiment, the batch start time is set to 1s to 2s, that is, the maximum batch start time Tv max Set to 2s, minimum batch duration Tv min Set to 1s.

[0116] Step S4.13: Generate a tracking sequence number according to the batch confidence of the current target, the batch confidence threshold CDT and the duration Tv of the video target trajectory, specifically including:

[0117] When the batch confidence of the current target is greater than the batch confidence threshold CDT, a tracking number is generated.

[0118] When the batch confidence of the current target is less than or equal to the batch confidence threshold CDT, and Tv≤Tv max When the target track duration Tv increases, the batch confidence threshold CDT decreases, and the batch confidence of the current target gradually exceeds the batch confidence threshold. At this time, the tracking number is generated. If Tv>Tv max , if the starting batch confidence of the current target is still less than or equal to the starting batch confidence threshold CDT, it indicates that it may not be the same video target, and the video target track is deleted. That is, when the starting batch confidence of the current target is less than or equal to the starting batch confidence threshold CDT, and Tv>Tv max , delete the video target track.

[0119] When the target track is a radar target track, a tracking sequence number is generated according to the target speed and duration of the radar target track, including:

[0120] Step S4.21: Calculate the average value of the target speed in each frame of the radar target trajectory, and use the average value as the starting speed of the current target.

[0121] Step S4.22: Calculate the starting speed threshold according to the duration of the radar target track. The specific formula is:

[0122]

[0123] Among them, ST represents the batch speed threshold, ST max Indicates the maximum batch speed threshold, ST min Indicates the minimum batch speed threshold, Tr max Indicates the maximum batch duration of the radar target track, Tr min In this embodiment, the starting speed threshold is set to 1m / s to 10m / s, that is, the maximum starting speed threshold ST max Set to 10m / s, minimum batch speed threshold ST min Set to 1m / s.

[0124] Step S4.23: Generate a tracking sequence number according to the batch speed of the current target, the batch speed threshold ST and the duration Tr of the radar target track, specifically including:

[0125] When the batch start speed of the current target is greater than the batch start speed threshold ST, a tracking number is generated.

[0126] When the batch start speed of the current target is less than or equal to the batch start speed threshold ST, and Tr≤Tr max When Tr>Tr, no tracking number is generated. As the duration Tr of the radar target track increases, the batch speed threshold ST decreases. The batch speed of the current target gradually exceeds the batch speed threshold. At this time, a tracking number is generated. max , if the starting speed of the current target is still less than or equal to the starting speed threshold ST, it indicates that it may not be the same radar target, and the radar target track is deleted. That is, when the starting speed of the current target is less than or equal to the starting speed threshold ST, and Tr>Tr max , delete the radar target track.

[0127] When the target trajectory is a radar target trajectory, a tracking number is generated according to the target confidence and duration of the radar target trajectory (such as steps S4.11 to S4.13), or a tracking number is generated according to the target speed and duration of the radar target trajectory (such as steps S4.21 to S4.23).

[0128] In order to avoid confusion caused by the generated tracking number being the same as the video target tracking number or the radar target tracking number, the generated tracking number belongs to a different numerical range from the video target tracking number or the radar target tracking number. For example, the video target tracking number is a value within the range of 1 to 3500, the radar target tracking number is a value within the range of 3501 to 7000, and the generated tracking number is a value within the range of 7001 to 9999.

[0129] The tracking number will not be generated in the following cases: (1) in the pixel coordinate system, when the target to be batched is within the video target frame corresponding to the existing tracking number; (2) in the radar coordinate system, when the distance between the target to be batched and the radar target position corresponding to the existing tracking number at the current moment is less than the set distance threshold.

[0130] Example 2

[0131] An embodiment of the present invention further provides a terminal device, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the multi-target tracking method described in Example 1.

[0132] Although not shown, the terminal device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage part to the random access memory (RAM). The processor can be a multi-core processor, or it can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for the operation of the terminal device are also stored. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0133] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0134] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the multi-target tracking method described in Embodiment 1 is implemented.

[0135] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0136] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0137] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which implements the multi-target tracking method as described in Example 1 when executed by a processor.

[0138] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A multi-target tracking method based on trajectory fusion, characterized in that: The tracking method comprises the following steps: Detect and track the video stream collected by the visual sensor to obtain the video target trajectory and video target tracking number; process and track the target echo signal detected by the radar to obtain the radar target trajectory and radar target tracking number; Performing frame rate alignment on the video target trajectory and the radar target trajectory according to the frame rate of the video stream; The video target trajectory and radar target trajectory after frame rate alignment are associated and fused to obtain the radar target trajectory; When the target trajectory is a video target trajectory, a tracking sequence number is generated according to the target confidence and duration of the video target trajectory; When the target track is a radar target track, a tracking sequence number is generated according to the target speed and duration of the radar target track; When the target track is a radar-sighted target track, a tracking number is generated according to the target confidence and duration of the radar-sighted target track, or a tracking number is generated according to the target speed and duration of the radar-sighted target track; According to the video target information obtained by detecting the video stream, a video target tracker is used to track the video target to obtain a video target trajectory; according to the radar target information obtained by processing the target echo signal, a radar target tracker is used to track the radar target to obtain a radar target trajectory; the specific process of the video target tracker or the radar target tracker performing target tracking includes: Cache the previous frame of target information, wherein the target information is video target information or radar target information; The target information of the current frame is used as the observation value, and the filtering algorithm is used to predict the target information of the previous frame to obtain the predicted value; Calculating a similarity matrix between the observed values ​​and the predicted values; The optimal matching matrix of the similarity matrix is ​​calculated by using the KM matching algorithm, and the optimal similarity between each observed value and the corresponding predicted value is determined according to the optimal matching matrix of the similarity matrix; When the optimal similarity exceeds the set similarity threshold, the observed value and the corresponding predicted value are the same target, and the target is successfully associated in two adjacent frames; when the optimal similarity is less than the set similarity threshold, the observed value is a new target appearing in the current frame, and the new target information is cached; When the target is successfully associated between consecutive frames, a target tracking number is assigned; when the target is failed to be associated between consecutive frames, the target information in the target tracker is deleted, wherein the target tracking number is a video target tracking number or a radar target tracking number, and the target tracker is a video target tracker or a radar target tracker.

2. The multi-target tracking method based on trajectory fusion according to claim 1, characterized in that: The frame rate of the video target track and the radar target track are aligned according to the frame rate of the video stream, specifically including: Obtain the timestamps of the video target trajectory and the radar target trajectory, wherein the timestamp of each video target in the video target trajectory is the moment when the target is detected on the corresponding frame image of the video stream, and the timestamp of each radar target in the radar target trajectory is the moment when the radar receives the corresponding target echo signal; Based on the frame rate of the video stream and the timestamp of each radar target in the radar target track, when the i-th frame of the radar target track has target information, the i-th frame of the radar target track is not interpolated; when the i-th frame of the radar target track does not have target information, the i-th frame of the radar target track is interpolated according to the target information of the i-1th frame of the radar target track.

3. The multi-target tracking method based on trajectory fusion according to claim 1, characterized in that: The KM matching algorithm is used to associate and fuse the video target trajectory and radar target trajectory after frame rate alignment, including: Calculate the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment; The KM matching algorithm is used to calculate the optimal matching matrix of the cost matrix, and the video target trajectory and the radar target trajectory are associated and fused according to the optimal matching matrix of the cost matrix to obtain the radar target trajectory.

4. The multi-target tracking method based on trajectory fusion according to claim 3 is characterized in that: Calculating the cost matrix between the video target trajectory and the radar target trajectory after frame rate alignment includes the following steps: Each video target track is converted into a video mapping target track in the radar coordinate system according to the conversion matrix between the radar coordinate system and the camera coordinate system; Calculate the cost trajectory sequence between the video mapping target trajectory and the radar target trajectory and the standard deviation σ of the cost trajectory sequence ij and mean μ ij , the specific calculation formula is: X ij =|V i -R j | Among them, X ij represents the cost trajectory sequence between the i-th video mapping target trajectory and the j-th radar target trajectory, X ij ={x ij1 ,x ij2 ,…,x ijk ,…,x ijn }, x ijk Represents the cost trajectory sequence X ij The kth value of V i represents the target trajectory of the i-th video mapping, Vi={vi1,vi2,…,vik,…,vin}, v ik Represents the coordinate information of the video target at the kth moment in the i-th video mapping target trajectory; R j represents the jth radar target track, R j = {r j1 ,r j2 ,…,r jk ,…,r jn }, r jk represents the coordinate information of the radar target at the kth moment in the jth radar target trajectory; n represents the length of the cost trajectory sequence or the video mapping target trajectory or the radar target trajectory; Take the cost trajectory sequence X ij In {σ ij -μ ij ,σ ij +μ ij } range as a valid trajectory sequence According to the cost trajectory sequence X ij And the effective trajectory sequence The cost value between the i-th video mapping target track and the j-th radar target track is calculated, and then the cost matrix between all video target tracks and all radar target tracks is obtained.

5. The multi-target tracking method based on trajectory fusion according to claim 4 is characterized in that: The specific calculation formula of the cost value between the i-th video mapping target track and the j-th radar target track is: Among them, C ij represents the cost between the i-th video mapping target track and the j-th radar target track, D t Indicates the set distance error threshold, P t represents the set probability threshold, Represents a valid trajectory sequence The average of the values ​​in .

6. The multi-target tracking method based on trajectory fusion according to any one of claims 1 to 5, characterized in that: The step of generating a tracking number according to the target confidence and duration of the target trajectory in the video specifically includes: Calculate the average value of the target confidence in the target trajectory of the video, and use the average value as the initial batch confidence of the current target; The confidence threshold for the start batch is calculated according to the duration of the target trajectory of the video. The specific formula is: Among them, CDT represents the confidence threshold of the batch, CDT max Indicates the maximum confidence threshold for the batch, CDT min Indicates the minimum confidence threshold for the batch, Tv max Indicates the maximum batch duration of the set video target track, Tv min Indicates the minimum batch duration of the set video target trajectory, and Tv indicates the duration of the video target trajectory; The tracking sequence number is generated according to the batch confidence of the current target, the batch confidence threshold CDT and the duration Tv of the video target trajectory.

7. The multi-target tracking method based on trajectory fusion according to claim 6, characterized in that: Generate a tracking number based on the current target's batch confidence, batch confidence threshold CDT, and the duration Tv of the video target track, including: When the batch confidence of the current target is greater than the batch confidence threshold CDT, a tracking number is generated; When the batch confidence of the current target is less than or equal to the batch confidence threshold CDT, and Tv≤Tv max When , no tracking number is generated; When the batch confidence of the current target is less than or equal to the batch confidence threshold CDT, and Tv>Tv max , delete the video target track.

8. The multi-target tracking method based on trajectory fusion according to any one of claims 1 to 5, characterized in that: The step of generating a tracking sequence number according to the target speed and duration of the radar target track specifically includes: Calculate the average value of the target speed in the radar target track, and use the average value as the starting speed of the current target; The batch starting speed threshold is calculated according to the duration of the radar target track. The specific formula is: Among them, ST represents the batch speed threshold, ST max Indicates the maximum batch speed threshold, ST min Indicates the minimum batch speed threshold, Tr max Indicates the maximum batch duration of the radar target track, Tr min Indicates the minimum batch duration of the radar target track, Tr indicates the duration of the radar target track; The tracking sequence number is generated according to the batch speed of the current target, the batch speed threshold ST and the duration Tr of the radar target track.

9. The multi-target tracking method based on trajectory fusion according to claim 8, characterized in that: The tracking sequence number is generated according to the batch speed of the current target, the batch speed threshold ST and the duration Tr of the radar target track, including: When the batch start speed of the current target is greater than the batch start speed threshold ST, a tracking number is generated; When the batch start speed of the current target is less than or equal to the batch start speed threshold ST, and Tr≤Tr max When , no tracking number is generated; When the current target batch speed is less than or equal to the batch speed threshold ST, and Tr>Tr max , delete the radar target track.

10. A terminal device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the multi-target tracking method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the multi-target tracking method according to any one of claims 1 to 9 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-target tracking method according to any one of claims 1 to 9 is implemented.

Images