Night vehicle cross-camera trajectory tracking method and device
Through the multi-scale fusion feature map and MCAM-YOLO network model, combined with Kalman filtering and IoU-ReID fusion information, the problem of cross-camera trajectory tracking of vehicles at night is solved, and accurate detection of vehicles at night and cross-camera trajectory tracking of vehicles is realized, which improves the management capabilities of smart traffic.
Patent Information
- Application Number
- CN202510600180.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-12
- Publication Date
- 2025-08-12
AI Technical Summary
The prior art has poor vehicle tracking accuracy in night environments and cannot track trajectory across cameras. Especially under traditional RGB cameras without infrared devices, the tracking capability has been greatly reduced and it is difficult to distinguish fast-moving vehicles.
The multi-scale fusion feature map is used to combine with the MCAM-YOLO network model. By generating multi-scale fusion of daytime and night data, the MCAM-YOLO network model is constructed, vehicle detection is performed, and trajectory matching is used using Kalman filtering, Hungarian algorithm and IoU-ReID fusion information to achieve trajectory tracking across cameras.
Accurate detection of vehicles and cross-camera tracking in night environments are achieved, which improves the ability to manage night vehicles in smart traffic, improves the accuracy and robustness of trajectory tracking, reduces the amount of calculation, and is suitable for real-time vehicle tracking.
Smart Images

Figure CN120471961A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of smart city vehicle management, and in particular to a method and device for tracking the trajectory of a vehicle across cameras at night. Background Art
[0002] Vehicle tracking is a key area in smart transportation. Currently, however, most vehicle tracking methods rely on satellite positioning and navigation or surveillance cameras. While satellite positioning and navigation are relatively mature, using surveillance cameras for tracking often presents the following challenges:
[0003] First, tracking can only be performed on a single camera and not across cameras, which greatly reduces the linkage capabilities between cameras in the road network;
[0004] Second, it can only handle well-lit daytime scenes. For traditional RGB cameras without infrared equipment, the tracking ability is greatly reduced at night.
[0005] Third, pedestrian re-identification cannot be simply applied to fast-moving vehicles under a single camera, because pedestrians are very different and easy to distinguish. However, for vehicles, there may be vehicles of the same model and appearance, which need to be distinguished separately.
[0006] Therefore, there is an urgent need to develop a method and device for tracking the trajectory of a vehicle across cameras at night. Summary of the Invention
[0007] In order to solve the problems of poor accuracy and difficulty in cross-camera tracking of existing vehicle tracking at night, the primary purpose of the present invention is to provide a nighttime vehicle cross-camera trajectory tracking method that can accurately detect vehicles in a nighttime environment and realize cross-camera trajectory tracking, greatly improving the ability of nighttime vehicle management in smart transportation.
[0008] To achieve the above object, the present invention adopts the following technical solution: a method for tracking the trajectory of a vehicle across cameras at night, the method comprising the following steps in sequence:
[0009] (1) Obtain video data information from multiple nighttime road cameras;
[0010] (2) Generate daytime data from the acquired video data information, perform multi-scale fusion of the daytime data and the nighttime data to obtain a multi-scale fusion feature map I;
[0011] (3) constructing an MCAM-YOLO network model, inputting the multi-scale fusion feature map I into the MCAM-YOLO network model, detecting each vehicle, and obtaining a detection result, which includes the location information of the vehicle and its detection frame;
[0012] (4) Numbering the detected vehicles and obtaining their trajectories through correlation matching;
[0013] (5) Compare the trajectories of the detected vehicles under different cameras, set the entry and exit points of the intersection, perform trajectory matching, and correct the number of the detected vehicles under each camera to obtain a unified global trajectory ID obtained by matching under different cameras;
[0014] (6) Based on the unified global trajectory ID, output a video stream with detection box, trajectory line, and vehicle information.
[0015] Step (2) specifically includes the following steps in order:
[0016] (2a) Convert the video data into frame-by-frame image data according to its frame rate be , and the image data f be Process according to the following formula to obtain the processed image data f a1 :
[0017] scale=min(640 / w,640 / h)
[0018] w'=w·scale,h'=h·scale
[0019]
[0020] Where scale is the scaling factor, w and h are f be The width and height, w' and h', are f a1 Width and height, Δh represents the width of the be Fill the top and bottom with 0 pixels.
[0021] Then, a reflection fill layer with a width of 3 is used to fill f a1 The tensor size is adjusted to (648, 648, 3) to obtain the feature map f a2 ; Use a convolution layer with a kernel size of 7x7 and a stride of 1, and adjust f a2 The tensor size is (640, 640, 3), and the feature map f is obtained a3 Normalize each pixel of each channel so that the mean of each channel is 0 and the standard deviation is 1. Then, pass the Leaky ReLU activation function layer to obtain the feature map f0. The tensor size of f0 is (640, 640, 3).
[0022] (2b) Using the Encoding Module f0 is encoded three times continuously to obtain the first encoding feature map f1, the second encoding feature map f2 and the third encoding feature map f3 respectively; the encoding module Contains a convolutional layer with a kernel size of 3x3 and a stride of 2, a normalization layer, and a Leaky ReLU activation function layer. The size of f1 is (320, 320, 64), the size of f2 is (160, 160, 128), and the size of f3 is (80, 80, 256);
[0023] (2c) Use the generation module G for f3 day Get the generated feature map f4: the generation module G day Contains 9 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer; the size of f4 is (80, 80, 256);
[0024] (2d) Using the decoding module Decode f4 three times in succession to obtain the first decoding feature map f5, the second decoding feature map f6 and the third decoding feature map f7 respectively; the decoding module It includes an upsampling layer with a scaling factor of 2, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, and a Leaky ReLU activation function layer; the size of f5 is (160, 160, 256), the size of f6 is (320, 320, 128), and the size of f7 is (640, 640, 64); f7 is converted to a night-time restored feature map f0' with a size of (640, 640, 3) through a convolution layer with a convolution kernel of 7x7 and a stride of 1;
[0025] (2e) For the training process, repeat steps (2b) to (2d) once, but modify the generation module G in step (2c) day To generate module G night , the generation module G night It contains 6 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer. The obtained layers are f1', f2', ..., f7', and f7' are converted to a daytime restoration feature map f0' of size (640, 640, 3). The discriminator D is introduced and the loss L = L1 + 10L2 + 5L3 is calculated, where:
[0026]
[0027] in, represents the expectation, L1 represents the loss of the discriminator D, y is the real daytime image; L2 represents the generation module G night The loss of L3 represents the generation module G day and G night The joint loss of y, D(y) represents the discrimination result of y, D(f0') represents the discrimination result of f0', G day (y) represents the result of generating y as daytime, G night (y) represents the result of generating night for y, G night (f0) represents the result of generating night for f0, G day (G night (y)) means using G for y night The result is then G day The results obtained;
[0028] (2f) Let f1, f2, f3, f5, f6, and f7 pass through a convolution layer with a 7x7 convolution kernel and a stride of 1, adjust f1, f2, f3, f5, f6, and f7 to the same size (640, 640, 64), and then splice the resized f1 and f7, f2 and f6, and f3 and f5 to obtain three feature maps f 17 ,f 26 ,f 35 , f 17 ,f 26 ,f 35 The size of both is (640, 640, 128), then let f 17 ,f 26 ,f 35 Then pass through a convolution layer with a 7x7 convolution kernel and a stride of 1 to adjust f 17 ,f 26 ,f 35 to the same size (640, 640, 64), and then splice the resized f 17 ,f 26 ,f 35 Into a whole, get the feature map f 172635 , f 172635 The size of f is (640, 640, 192), then let f 172635 Then, through a convolution layer with a 7x7 convolution kernel and a stride of 1, a multi-scale fusion feature map I is obtained, and the size of the multi-scale fusion feature map I is (640, 640, 3).
[0029] Step (3) specifically includes the following steps in order:
[0030] (3a) Normalizing the multi-scale fusion feature map I to obtain the normalized data I0;
[0031] (3b) Constructing an MCAM-YOLO network model and processing the normalized data I0: using the YOLOv12 network as the basic network, the YOLOv12 network includes a backbone network, a neck network, and a detection head, and replacing the A2C2f module in the backbone network with an MCAC3k module; the MCAC3k module includes a convolution layer with a convolution kernel size of 1x1 and a stride of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, an average pooling layer, and a bias pooling layer;
[0032] (3c) Output the detection results from the detection head.
[0033] Step (4) specifically includes the following steps in order:
[0034] (4a) Divide the detection box into high confidence detection boxes Bs according to the confidence level high With low confidence detection box Bs low Two categories, the high confidence detection box Bs high The image in the image is used for ReID feature extraction, and the extracted feature information is f i k ; Use the FastReID model with ResNeSt50 as the backbone to extract the location information of the vehicle in the detection frame and compare the location information with the high confidence detection frame Bs high Embed the appearance information in:
[0035]
[0036] Where, F i k is the appearance motion feature vector obtained by embedding the appearance and position of the current matching detection, and are the detection box parameters of the previous frame and this frame of the i-th trajectory, β1 and β2 are the weights of the appearance and position information embedding, respectively, IoU(b i ,b i k ) is the intersection-over-union ratio between the current detection frame and the matching frame at the previous moment:
[0037]
[0038] Where (x1, y1) is the bounding box The coordinates of the lower left corner, (x2, y2) is the bounding box The coordinates of the upper right corner; (x 1' ,y 1' ) is the bounding box The coordinates of the lower left corner, (x 2' ,y 2' ) is the bounding box The coordinates of the upper right corner, (x int1 ,y int1 ) is the bounding box With bounding box The coordinates of the lower left corner of the intersection area, (x int2 ,y int2 ) is the coordinate of the upper right corner of the intersection area of bounding box b1 and bounding box b2;
[0039] The exponential moving average mechanism is used to update the matching trajectory appearance position information state e of the i-th trajectory at the k-th frame i k :
[0040]
[0041] Among them, α = 0.9 is the momentum term;
[0042] Position information status in average trajectory appearance and F i k Match between them and measure the cosine similarity;
[0043] (4b) Perform camera motion compensation and use OpenCV's global motion estimation technology to represent background motion. First, extract image key points, then use sparse optical flow to perform feature tracking based on translation-based local outlier suppression. First, use the random sampling consensus algorithm to calculate the affine transformation matrix. The predicted bounding box is then transformed from the coordinate system of the k-1 frame to the coordinate system of the next frame k using the affine transformation matrix. The translation part of the affine transformation matrix only affects the center position of the bounding box, while the other part affects all state vectors and noise matrices. Camera motion compensation is performed by the following equation:
[0044]
[0045] in, Represents the motion transformation matrix of the camera from the k-1th moment to the kth moment, M 2x2 represents the linear transformation matrix of the camera from time k-1 to time k, M is the scaling and rotation matrix; T 2x1 Represents the translation transformation matrix of the camera from the k-1th moment to the kth moment, is the state vector dimension transformation matrix, is the bounding box coordinate transformation matrix;
[0046] Then combine camera motion compensation with Kalman filtering:
[0047]
[0048] Where, They are the predicted state vectors of the Kalman filter at time k before and after camera motion compensation, P k|k-1 , P′ k|k-1 are the prediction covariance matrices of the Kalman filter before and after correction, respectively;
[0049] (4c) F i k and Combine and associate the first-level trajectory with the detection frame; the cost matrix used in the association process is C, which is combined with the appearance cosine matrix d embedded in the motion information cos and IoU distance matrix d iou ;
[0050] d cos Elements is the appearance motion feature vector F in the i-th track i and the appearance motion feature vector F of the jth trajectory j The cosine distance between:
[0051]
[0052] Where, the IoU distance matrix d iou Elements in The IoU distance between the predicted bounding box i and the detected bounding box j of the trajectory is represented as IoU(b i ,b j ):
[0053]
[0054] Among them, b i is the coordinate information of the i-th predicted bounding box, b j is the coordinate information of the jth detection bounding box;
[0055] Then, the minimum value of each element of the matrix is used as the final value of the cost matrix C to fuse the IoU information with the ReID information. The resulting cost matrix C of the fused IoU-ReID information is as follows:
[0056]
[0057] Among them, C i,j is the (i,j) element of the cost matrix C, is the IoU distance between the i-th predicted bounding box and the j-th detected bounding box of the track; is the appearance feature F embedded with motion information i and the newly detected appearance feature F embedded with motion informationj The cosine distance between for Corrected distance; θ iou is the proximity threshold, set to 0.5; θ emb is the appearance threshold;
[0058] (4d) According to C i,j The Hungarian algorithm is used for matching. The detected target is associated with the existing tracking trajectory. The set of successfully matched trajectories is T o1 ;
[0059] During the matching process, the Static Sensitive Association (SSA) strategy is used. When the vehicle detection frame is detected to be stationary, the position and appearance state information of the corresponding track will be saved until the vehicle starts to activate it and participate in the matching process.
[0060] (4e) Track the trajectory bidirectionally: first, forward tracking is performed, running the tracker from front to back on the video frame, generating a trajectory for each target based on the detected target and the trajectory information of the previous frame; then, backward tracking is performed, running the tracker forward from the last frame of the video, so that targets close to the camera are easier to detect and the trajectory is initialized; finally, the tracks are merged to obtain a complete trajectory;
[0061] (4f) The low confidence detection box Bs low Perform secondary matching with the unmatched trajectories in step (4d). This matching is based on IoU. First, calculate the IoU matrix: calculate the prediction box updated by Kalman filter for each trajectory With Bs low The intersection-over-union ratio between the detection boxes in the , constitutes the matrix IoU, the matrix elements are:
[0062]
[0063] in, is the detection box predicted by trajectory i using Kalman filtering at time t+1, B j,t It's Bs low The j-th detection box in ;
[0064] Then the Hungarian algorithm is used to match the intersection-over-union matrix as the cost matrix IoU. The set of successfully matched trajectories is T o2 ;
[0065] (4g) Relink the trajectory: According to T o1 With T o2 The starting and ending positions of the trajectory in the , if the starting or ending coordinates of the trajectory fall within the area of the pre-calibrated abnormality, it is classified into the abnormal trajectory set T e , for the set T eRelink the trajectories in
[0066] Use f i k Construct the cosine distance matrix C f , take the set T e The i-th trajectory T in ei and the jth trajectory T ej , C f Element C fi,j For these two trajectories T ei With T ej The cosine distance of the appearance feature:
[0067]
[0068] Then use the greedy algorithm to merge;
[0069] (4h) T o1 With T o2 To manage and update the trajectory in , first update the state of the Kalman filter, and then update the appearance information in the trajectory.
[0070] Step (5) specifically includes the following steps in order:
[0071] (5a) Divide the road under different cameras into entrance and exit areas, where the camera set is N is the total number of cameras. The camera entrance and exit areas are defined and matched using the adjacency matrix A∈{0,1} N×N , for two cameras C ami with C amj ,for A ij =1 means C ami The exit area and C amj The entrance area is adjacent to the
[0072] (5b) Using time information and entrance and exit matching information, filter the trajectories under different cameras; each camera C ami The trajectory collection in, Indicates camera C ami The trajectory of T i k Timestamp of entering the camera area, Indicates camera C ami The trajectory of T i k Timestamp of leaving the camera area, F i k Represents the appearance and position embedding of the matched detection, and the appearance motion feature vector, R ik Indicates area information; take camera C ami Any trajectory T in i m With camera C amj Any trajectory T in j n , retain two trajectories T that satisfy the following time and space consistency constraints among all trajectories i m and
[0073]
[0074] in, T is the trajectory i m Enter camera C ami Regional timestamp, T is the trajectory j n Camera C amj Timestamp, Δt max is the maximum allowed time difference, F i m T is the trajectory i m The appearance motion feature vector, F i n T is the trajectory i n The appearance motion feature vector, θ feat is the feature similarity threshold;
[0075] Get the trajectory set T that meets the entrance, exit and time constraints after filtering s ';
[0076] (5c) for T s Each of the two different trajectories T m 'With T n ', using F i k , calculate the similarity matrix S, K is T s 'The number of trajectories in the
[0077] The nth element in the mth row of the matrix S mn The formula is:
[0078]
[0079] Where, φ: It represents the function of deep feature extraction using feature extraction network;
[0080] (5d) Use the Hungarian algorithm to find the most fitting trajectory and save the result in the binary matching matrix X:
[0081]
[0082] Then use the Hungarian algorithm to solve the element X mn is the element of the binary matching matrix; when X mn When the value is 1, it means the trajectory T i m and trajectory Matching; the final matching results are saved as matching matrix X * ;
[0083] (5e) According to the matched trajectory, correct the trajectory T' successfully matched in step (5d), and use the matched trajectory to correct T i k and Update the logo:
[0084]
[0085] Among them, G(x) is the global ID allocation function, is the matching matrix X * Elements, Indicates camera C amj The nth trajectory in T i m Indicates camera C ami The mth trajectory in For camera C amj The unified global trajectory ID of the nth trajectory in .
[0086] In step (3b), the processing of the MCAC3k module includes the following steps:
[0087] (3b1) Input I0 into the MCAM-YOLO network model to obtain the input I1 of the MCAC3k module. Then, a convolution layer with a convolution kernel size of 1x1 and a stride of 1 is used to adjust the number of channels to half of I1. Then, a convolution layer with a convolution kernel size of 3x3 and a stride of 1 is used to extract information to obtain the feature map I2.
[0088] (3b2) Horizontal processing is performed on the feature map I2; the horizontal processing includes exchanging the channel dimension and the height dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3w ;
[0089] (3b3) Processing the feature map I2 in the vertical direction; the vertical processing includes exchanging the channel dimension and the width dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling; obtaining the feature map I 3h ;
[0090] (3b4) Processing the feature map I2 in the channel direction; the channel direction processing includes performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3c ;
[0091] (3b5) for feature map I 3w , I 3h , I 3c Add and average to obtain the processed feature map I4, then add the residual of I4 and I2 to obtain the feature map I5; then let I5 pass through a convolution layer with a convolution kernel size of 1x1 and a stride of 1 to obtain the feature map I6, and finally, I6 is residually connected to I1 to output the feature map I7.
[0092] In step (4g), the merging is performed using a greedy algorithm, and the process includes the following steps:
[0093] (4g1) Arrange the fracture tracks according to the starting time;
[0094] (4g2) Starting from the first trajectory, find the trajectory with the smallest cosine distance to its ReID feature among the remaining trajectories;
[0095] (4g3) If the minimum distance is less than the set threshold, merge the two trajectories and update the trajectory set; otherwise, keep the current trajectory alone;
[0096] (4g4) Repeat steps (4g1) to (4g3) until all trajectories are processed.
[0097] Get the merged trajectory set.
[0098] Another object of the present invention is to provide an electronic device, comprising:
[0099] processor; and
[0100] A memory stores computer program instructions, which, when executed by the processor, enable the processor to execute the method for tracking the trajectory of a vehicle across cameras at night.
[0101] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is enabled to execute the method for tracking the trajectory of a vehicle across cameras at night as described above.
[0102] It can be seen from the above technical solution that the beneficial effects of the present invention are as follows: First, the present invention realizes cross-camera trajectory tracking of vehicles at night. Previous vehicle trajectory tracking methods are not effective at night, especially under traditional RGB cameras without infrared equipment, and the tracking ability is greatly reduced. The present invention generates multi-scale fusion of daytime data and nighttime data, and combines it with the MCAM-YOLO network model. It can accurately detect vehicles in nighttime environments and realize cross-camera trajectory tracking, effectively making up for the shortcomings of existing technologies in nighttime cross-camera tracking and greatly improving the ability of nighttime vehicle management in smart transportation; Second, the present invention improves the accuracy and robustness of trajectory tracking. On the basis of traditional Kalman filtering and Hungarian algorithm, the present invention adds IoU-ReID fusion information, stationary sensitive association strategy, trajectory relinking strategy, and bidirectional tracking strategy, so that trajectory tracking can still maintain high accuracy and stability in complex traffic scenes and vehicle motion states; Third, the present invention improves the recognition speed of the vehicle detection model. While maintaining the attention mechanism, the number of model parameters and the amount of calculation are greatly reduced, so that the system can process a large amount of video data in a relatively short time, which is suitable for real-time vehicle trajectory tracking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0103] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION
[0104] like Figure 1 As shown, a method for tracking a vehicle's trajectory across cameras at night includes the following steps in sequence:
[0105] (1) Obtain video data information from multiple nighttime road cameras;
[0106] (2) Generate daytime data from the acquired video data information, perform multi-scale fusion of the daytime data and the nighttime data to obtain a multi-scale fusion feature map I;
[0107] (3) constructing an MCAM-YOLO network model, inputting the multi-scale fusion feature map I into the MCAM-YOLO network model, detecting each vehicle, and obtaining a detection result, which includes the location information of the vehicle and its detection frame;
[0108] (4) Numbering the detected vehicles and obtaining their trajectories through correlation matching;
[0109] (5) Compare the trajectories of the detected vehicles under different cameras, set the entry and exit points of the intersection, perform trajectory matching, and correct the number of the detected vehicles under each camera to obtain a unified global trajectory ID obtained by matching under different cameras;
[0110] (6) Based on the unified global trajectory ID, output a video stream with detection box, trajectory line, and vehicle information.
[0111] Step (2) specifically includes the following steps in order:
[0112] (2a) Convert the video data into frame-by-frame image data according to its frame rate be , and the image data f be Process according to the following formula to obtain the processed image data f a1 :
[0113] scale=min(640 / w,640 / h)
[0114] w'=w·scale,h'=h·scale
[0115]
[0116] Where scale is the scaling factor, w and h are f be The width and height, w' and h', are f a1 Width and height, Δh represents the width of the be Fill the top and bottom with 0 pixels.
[0117] Then, a reflection fill layer with a width of 3 is used to fill f a1 The tensor size is adjusted to (648, 648, 3) to obtain the feature map f a2 ; Use a convolution layer with a kernel size of 7x7 and a stride of 1, and adjust f a2 The tensor size is (640, 640, 3), and the feature map f is obtained a3 Normalize each pixel of each channel so that the mean of each channel is 0 and the standard deviation is 1. Then, pass the Leaky ReLU activation function layer to obtain the feature map f0. The tensor size of f0 is (640, 640, 3).
[0118] (2b) Using the Encoding Module f0 is encoded three times continuously to obtain the first encoding feature map f1, the second encoding feature map f2 and the third encoding feature map f3 respectively; the encoding module Contains a convolutional layer with a kernel size of 3x3 and a stride of 2, a normalization layer, and a Leaky ReLU activation function layer. The size of f1 is (320, 320, 64), the size of f2 is (160, 160, 128), and the size of f3 is (80, 80, 256);
[0119] (2c) Use the generation module G for f3 day Get the generated feature map f4: the generation module G day Contains 9 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer; the size of f4 is (80, 80, 256);
[0120] (2d) Using the decoding module Decode f4 three times in succession to obtain the first decoding feature map f5, the second decoding feature map f6 and the third decoding feature map f7 respectively; the decoding module It includes an upsampling layer with a scaling factor of 2, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, and a Leaky ReLU activation function layer; the size of f5 is (160, 160, 256), the size of f6 is (320, 320, 128), and the size of f7 is (640, 640, 64); f7 is converted to a night-time restored feature map f0' with a size of (640, 640, 3) through a convolution layer with a convolution kernel of 7x7 and a stride of 1;
[0121] (2e) For the training process, repeat steps (2b) to (2d) once, but modify the generation module G in step (2c) day To generate module G night , the generation module G night It contains 6 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer. The obtained layers are f1', f2', ..., f7', and f7' are converted to a daytime restoration feature map f0' of size (640, 640, 3). The discriminator D is introduced and the loss L = L1 + 10L2 + 5L3 is calculated, where:
[0122]
[0123] in, represents the expectation, L1 represents the loss of the discriminator D, y is the real daytime image; L2 represents the generation module G night The loss of L3 represents the generation module G day and G night The joint loss of y, D(y) represents the discrimination result of y, D(f0') represents the discrimination result of f0', G day (y) represents the result of generating y as daytime, G night (y) represents the result of generating night for y, G night (f0) represents the result of generating night for f0, G day (G night (y)) means using G for y night The result is then G day The results obtained;
[0124] (2f) Let f1, f2, f3, f5, f6, and f7 pass through a convolution layer with a 7x7 convolution kernel and a stride of 1, adjust f1, f2, f3, f5, f6, and f7 to the same size (640, 640, 64), and then splice the resized f1 and f7, f2 and f6, and f3 and f5 to obtain three feature maps f 17 ,f 26 ,f 35 , f 17 ,f 26 ,f 35 The size of both is (640, 640, 128), then let f 17 ,f 26 ,f 35 Then pass through a convolution layer with a 7x7 convolution kernel and a stride of 1 to adjust f 17 ,f 26 ,f 35 to the same size (640, 640, 64), and then splice the resized f 17 ,f 26 ,f 35 Into a whole, get the feature map f 172635 , f 172635 The size of f is (640, 640, 192), then let f 172635 Then, through a convolution layer with a 7x7 convolution kernel and a stride of 1, a multi-scale fusion feature map I is obtained, and the size of the multi-scale fusion feature map I is (640, 640, 3).
[0125] Step (3) specifically includes the following steps in order:
[0126] (3a) Normalizing the multi-scale fusion feature map I to obtain the normalized data I0;
[0127] (3b) Constructing an MCAM-YOLO network model and processing the normalized data I0: using the YOLOv12 network as the basic network, the YOLOv12 network includes a backbone network, a neck network, and a detection head, and replacing the A2C2f module in the backbone network with an MCAC3k module; the MCAC3k module includes a convolution layer with a convolution kernel size of 1x1 and a stride of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, an average pooling layer, and a bias pooling layer;
[0128] (3c) Output the detection results from the detection head.
[0129] Step (4) specifically includes the following steps in order:
[0130] (4a) Divide the detection box into high confidence detection boxes Bs according to the confidence level high With low confidence detection box Bs low Two categories, the high confidence detection box Bs high The image in the image is used for ReID feature extraction, and the extracted feature information is f i k ; Use the FastReID model with ResNeSt50 as the backbone to extract the location information of the vehicle in the detection frame and compare the location information with the high confidence detection frame Bs high Embed the appearance information in:
[0131]
[0132] Where, F i k is the appearance motion feature vector obtained by embedding the appearance and position of the current matching detection, and are the detection box parameters of the previous frame and this frame of the i-th trajectory, β1 and β2 are the weights of the appearance and position information embedding, respectively. It is the intersection-over-union ratio between the current detection box and the matching box at the previous moment:
[0133]
[0134] Where (x1, y1) is the bounding box The coordinates of the lower left corner, (x2, y2) is the bounding box The coordinates of the upper right corner; (x 1' ,y 1' ) is the bounding box The coordinates of the lower left corner, (x 2' ,y 2' ) is the bounding box The coordinates of the upper right corner, (xint1 ,y int1 ) is the bounding box With bounding box The coordinates of the lower left corner of the intersection area, (x int2 ,y int2 ) is the coordinate of the upper right corner of the intersection area of bounding box b1 and bounding box b2;
[0135] The exponential moving average mechanism is used to update the matching trajectory appearance position information state e of the i-th trajectory at the k-th frame i k :
[0136]
[0137] Among them, α = 0.9 is the momentum term;
[0138] Position information status in average trajectory appearance and F i k Match between them and measure the cosine similarity;
[0139] (4b) Perform camera motion compensation and use OpenCV's global motion estimation technology to represent background motion. First, extract image key points, then use sparse optical flow to perform feature tracking based on translation-based local outlier suppression. First, use the random sampling consensus algorithm to calculate the affine transformation matrix. The predicted bounding box is then transformed from the coordinate system of the k-1 frame to the coordinate system of the next frame k using the affine transformation matrix. The translation part of the affine transformation matrix only affects the center position of the bounding box, while the other part affects all state vectors and noise matrices. Camera motion compensation is performed by the following equation:
[0140]
[0141] in, Represents the motion transformation matrix of the camera from the k-1th moment to the kth moment, M 2x2 represents the linear transformation matrix of the camera from time k-1 to time k, M is the scaling and rotation matrix; T 2x1 Represents the translation transformation matrix of the camera from the k-1th moment to the kth moment, is the state vector dimension transformation matrix, is the bounding box coordinate transformation matrix;
[0142] Then combine camera motion compensation with Kalman filtering:
[0143]
[0144] Where, They are the predicted state vectors of the Kalman filter at time k before and after camera motion compensation, P k|k-1 , P′ k|k-1 are the prediction covariance matrices of the Kalman filter before and after correction, respectively;
[0145] (4c) F i k and Combine and associate the first-level trajectory with the detection frame; the cost matrix used in the association process is C, which is combined with the appearance cosine matrix d embedded in the motion information cos and IoU distance matrix d iou ;
[0146] d cos Elements is the appearance motion feature vector F in the i-th track i and the appearance motion feature vector F of the jth trajectory j The cosine distance between:
[0147]
[0148] Where, the IoU distance matrix d iou Elements in The IoU distance between the predicted bounding box i and the detected bounding box j of the trajectory is represented as IoU(b i ,b j ):
[0149]
[0150] Among them, b i is the coordinate information of the i-th predicted bounding box, b j is the coordinate information of the jth detection bounding box;
[0151] Then, the minimum value of each element of the matrix is used as the final value of the cost matrix C to fuse the IoU information with the ReID information. The resulting cost matrix C of the fused IoU-ReID information is as follows:
[0152]
[0153] Among them, C i,j is the (i,j) element of the cost matrix C, is the IoU distance between the i-th predicted bounding box and the j-th detected bounding box of the track; is the appearance feature F embedded with motion information i and the newly detected appearance feature F embedded with motion information j The cosine distance between for Corrected distance; θ iou is the proximity threshold, set to 0.5; θ emb is the appearance threshold;
[0154] (4d) According to C i,j The Hungarian algorithm is used for matching. The detected target is associated with the existing tracking trajectory. The set of successfully matched trajectories is T o1 ;
[0155] During the matching process, a stationary-sensitive association (SSA) strategy is used. When a vehicle detection frame is detected as stationary, the corresponding track's position and appearance state information is saved until the vehicle starts, at which point it is activated for the matching process. Given that vehicles often stop and start at intersections in real-world applications, when a track and its surrounding tracks are stationary, the highest-scoring detection result is retained as its position information, effectively addressing the problem of missed detections due to occlusion. To further enhance robustness against nonlinear motion, a smoothed Mahalanobis distance method is employed to avoid data mutations during nonlinear motion, ensuring system stability and accuracy.
[0156] A trajectory relinking (TRL) strategy is also used. In severe occlusion situations, the vehicle's track ID switches after being obstructed. Trajectories that end or begin in the middle of the scene are filtered out. A greedy algorithm is then used to merge these fragmented trajectories based on the cosine distance of their ReID features. This addresses the issue of the Kalman filter failing to accurately predict the motion state due to the rapid change in speed at vehicle start-up.
[0157] (4e) Track the trajectory bidirectionally: first, forward tracking is performed, running the tracker from front to back on the video frame, generating a trajectory for each target based on the detected target and the trajectory information of the previous frame; then, backward tracking is performed, running the tracker forward from the last frame of the video, so that targets close to the camera are easier to detect and the trajectory is initialized; finally, the tracks are merged to obtain a complete trajectory;
[0158] (4f) The low confidence detection box Bs low Perform secondary matching with the unmatched trajectories in step (4d). This matching is based on IoU, so the computational complexity is low. This can reduce redundant computations. First, calculate the IoU matrix: Calculate the prediction box updated by the Kalman filter for each trajectory. With Bs low The intersection-over-union ratio between the detection boxes in the , constitutes the matrix IoU, the matrix elements are:
[0159]
[0160] in, is the detection box predicted by trajectory i using Kalman filtering at time t+1, B j,t It's Bs low The j-th detection box in ;
[0161] Then the Hungarian algorithm is used to match the intersection-over-union matrix as the cost matrix IoU. The set of successfully matched trajectories is T o2 ;
[0162] (4g) Relink the trajectory: According to T o1 With T o2 The starting and ending positions of the trajectory in the , if the starting or ending coordinates of the trajectory fall within the area of the pre-calibrated abnormality, it is classified into the abnormal trajectory set T e , for the set T e Relink the trajectories in
[0163] Use f i k Construct the cosine distance matrix C f , take the set T e The i-th trajectory T in ei and the jth trajectory T ej , C f Element C fi,j For these two trajectories T ei With T ej The cosine distance of the appearance feature:
[0164]
[0165] Then use the greedy algorithm to merge;
[0166] (4h) T o1 With T o2 To manage and update the trajectory in , first update the state of the Kalman filter, and then update the appearance information in the trajectory.
[0167] Step (5) specifically includes the following steps in order:
[0168] (5a) Divide the road under different cameras into entrance and exit areas, where the camera set is N is the total number of cameras. The entrance and exit areas of the cameras are defined and matched using the adjacency matrix A∈{0,1} N×N , for two cameras C ami with C amj ,for A ij =1 means C ami The exit area and C amj The entrance area is adjacent to the
[0169] (5b) Using time information and entrance and exit matching information, filter the trajectories under different cameras; each camera C ami The trajectory collection in, Indicates camera C ami The trajectory of T i k Timestamp of entering the camera area, Indicates camera C ami The trajectory of T i k Timestamp of leaving the camera area, F i k Represents the appearance and position embedding of the matched detection, and the appearance motion feature vector, R i k Indicates area information; take camera C ami Any trajectory T in i m With camera C amj Any trajectory in Among all the trajectories, keep two trajectories T that satisfy the following time and space consistency constraints i m and
[0170]
[0171] in, T is the trajectory i m Enter camera C ami Regional timestamp, For the trajectory Camera C amj Timestamp, Δt max is the maximum allowed time difference, F i m T is the trajectory i m The appearance motion feature vector, F i n T is the trajectory i n The appearance motion feature vector, θ feat is the feature similarity threshold;
[0172] Get the trajectory set T that meets the entrance, exit and time constraints after filtering s ';
[0173] (5c) for T s Each of the two different trajectories T m 'With T n ', using Fi k , calculate the similarity matrix S, K is T s 'The number of trajectories in the
[0174] The nth element in the mth row of the matrix S mn The formula is:
[0175]
[0176] Where, φ: It represents the function of deep feature extraction using feature extraction network;
[0177] (5d) Use the Hungarian algorithm to find the most fitting trajectory and save the result in the binary matching matrix X:
[0178]
[0179] Then use the Hungarian algorithm to solve the element X mn is the element of the binary matching matrix; when X mn When the value is 1, it means the trajectory T i m and trajectory Matching; the final matching results are saved as matching matrix X * ;
[0180] (5e) According to the matched trajectory, correct the trajectory T' successfully matched in step (5d), and use the matched trajectory to correct T i k and Update the logo:
[0181]
[0182] Among them, G(x) is the global ID allocation function, is the matching matrix X * Elements, Indicates camera C amj The nth trajectory in T i m Indicates camera C ami The mth trajectory in For camera C amj The unified global trajectory ID of the nth trajectory in .
[0183] In step (3b), the processing of the MCAC3k module includes the following steps:
[0184] (3b1) Input I0 into the MCAM-YOLO network model to obtain the input I1 of the MCAC3k module. Then, a convolution layer with a convolution kernel size of 1x1 and a stride of 1 is used to adjust the number of channels to half of I1. Then, a convolution layer with a convolution kernel size of 3x3 and a stride of 1 is used to extract information to obtain the feature map I2.
[0185] (3b2) Horizontal processing is performed on the feature map I2; the horizontal processing includes exchanging the channel dimension and the height dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3w ;
[0186] (3b3) Processing the feature map I2 in the vertical direction; the vertical processing includes exchanging the channel dimension and the width dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling; obtaining the feature map I 3h ;
[0187] (3b4) Processing the feature map I2 in the channel direction; the channel direction processing includes performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3c ;
[0188] (3b5) for feature map I 3w , I 3h , I 3c Add and average to obtain the processed feature map I4, then add the residual of I4 and I2 to obtain the feature map I5; then let I5 pass through a convolution layer with a convolution kernel size of 1x1 and a stride of 1 to obtain the feature map I6, and finally, I6 is residually connected to I1 to output the feature map I7.
[0189] In step (4g), the merging is performed using a greedy algorithm, and the process includes the following steps:
[0190] (4g1) Arrange the fracture tracks according to the starting time;
[0191] (4g2) Starting from the first trajectory, find the trajectory with the smallest cosine distance to its ReID feature among the remaining trajectories;
[0192] (4g3) If the minimum distance is less than the set threshold, merge the two trajectories and update the trajectory set; otherwise, keep the current trajectory alone;
[0193] (4g4) Repeat steps (4g1) to (4g3) until all trajectories are processed.
[0194] Get the merged trajectory set.
[0195] In summary, the present invention realizes cross-camera trajectory tracking of vehicles at night. Previous vehicle trajectory tracking methods are not effective at night, especially under traditional RGB cameras without infrared equipment, the tracking ability is greatly reduced. The present invention generates multi-scale fusion of daytime data and nighttime data, and combines it with the MCAM-YOLO network model. It can accurately detect vehicles in a night environment and realize cross-camera trajectory tracking, effectively making up for the shortcomings of the existing technology in cross-camera tracking at night, and greatly improving the ability of night vehicle management in smart transportation; the present invention improves the accuracy and robustness of trajectory tracking. On the basis of traditional Kalman filtering and Hungarian algorithm, the present invention adds IoU-ReID fusion information, static sensitive association strategy, trajectory relinking strategy, and bidirectional tracking strategy, so that trajectory tracking can still maintain high accuracy and stability in complex traffic scenes and vehicle motion states; the present invention improves the recognition speed of the vehicle detection model, and greatly reduces the number of parameters and calculation amount of the model while maintaining the attention mechanism, so that the system can process a large amount of video data in a relatively short time, which is suitable for real-time vehicle trajectory tracking scenarios.
[0196] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for tracking vehicle trajectories across cameras at night, characterized by: The method comprises the following steps in sequence: (1) Obtain video data information from multiple nighttime road cameras; (2) Generate daytime data from the acquired video data information, perform multi-scale fusion of the daytime data and the nighttime data to obtain a multi-scale fusion feature map I; (3) constructing an MCAM-YOLO network model, inputting the multi-scale fusion feature map I into the MCAM-YOLO network model, detecting each vehicle, and obtaining a detection result, which includes the location information of the vehicle and its detection frame; (4) Numbering the detected vehicles and obtaining their trajectories through correlation matching; (5) Compare the trajectories of the detected vehicles under different cameras, set the entry and exit points of the intersection, perform trajectory matching, and correct the number of the detected vehicles under each camera to obtain a unified global trajectory ID obtained by matching under different cameras; (6) Based on the unified global trajectory ID, output a video stream with detection box, trajectory line, and vehicle information.
2. The method for tracking a vehicle's trajectory across cameras at night according to claim 1, characterized in that: Step (2) specifically includes the following steps in order: (2a) Convert the video data into frame-by-frame image data according to its frame rate be , and the image data f be Process according to the following formula to obtain the processed image data f a1 : scale=min(640 / w,640 / h) w'=w·scale,h'=h·scale Where scale is the scaling factor, w and h are f be The width and height, w' and h', are f a1 Width and height, Δh represents the width of the be Fill the top and bottom with 0 pixels. Then, a reflection fill layer with a width of 3 is used to fill f a1 The tensor size is adjusted to (648, 648, 3) to obtain the feature map f a2 ; Use a convolution layer with a kernel size of 7x7 and a stride of 1, and adjust f a2 The tensor size is (640, 640, 3), and the feature map f is obtained a3 Normalize each pixel of each channel so that the mean of each channel is 0 and the standard deviation is 1. Then, pass the Leaky ReLU activation function layer to obtain the feature map f0. The tensor size of f0 is (640, 640, 3). (2b) Using the Encoding Module f0 is encoded three times continuously to obtain the first encoding feature map f1, the second encoding feature map f2 and the third encoding feature map f3 respectively; the encoding module Contains a convolutional layer with a kernel size of 3x3 and a stride of 2, a normalization layer, and a Leaky ReLU activation function layer. The size of f1 is (320, 320, 64), the size of f2 is (160, 160, 128), and the size of f3 is (80, 80, 256); (2c) Use the generation module G for f3 day Get the generated feature map f4: the generation module G day Contains 9 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolutional layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer; the size of f4 is (80, 80, 256); (2d) Using the decoding module Decode f4 three times in succession to obtain the first decoding feature map f5, the second decoding feature map f6 and the third decoding feature map f7 respectively; the decoding module It includes an upsampling layer with a scaling factor of 2, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, and a Leaky ReLU activation function layer; the size of f5 is (160, 160, 256), the size of f6 is (320, 320, 128), and the size of f7 is (640, 640, 64); f7 is converted to a night-time restored feature map f0' with a size of (640, 640, 3) through a convolution layer with a convolution kernel of 7x7 and a stride of 1; (2e) For the training process, repeat steps (2b) to (2d) once, but modify the generation module G in step (2c) day To generate module G night , the generation module G night It contains 6 consecutive residual layers with the same structure. Each residual layer includes, in order, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, a normalization layer, a Leaky ReLU activation function layer, a reflection padding layer with a width of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, and a normalization layer. The obtained layers are f1', f2', ..., f7', and f7' are converted to a daytime restoration feature map f0' of size (640, 640, 3). The discriminator D is introduced and the loss L = L1 + 10L2 + 5L3 is calculated, where: in, represents the expectation, L1 represents the loss of the discriminator D, y is the real daytime image; L2 represents the generation module G night The loss of L3 represents the generation module G day and G night The joint loss of y, D(y) represents the discrimination result of y, D(f0') represents the discrimination result of f0', G day (y) represents the result of generating y as daytime, G night (y) represents the result of generating night for y, G night (f0) represents the result of generating night for f0, G day (G night (y)) means using G for y night The result is then G day The results obtained; (2f) Let f1, f2, f3, f5, f6, and f7 pass through a convolution layer with a 7x7 convolution kernel and a stride of 1, adjust f1, f2, f3, f5, f6, and f7 to the same size (640, 640, 64), and then splice the resized f1 and f7, f2 and f6, and f3 and f5 to obtain three feature maps f 17 ,f 26 ,f 35 , f 17 ,f 26 ,f 35 The size of both is (640, 640, 128), then let f 17 ,f 26 ,f 35 Then pass through a convolution layer with a 7x7 convolution kernel and a stride of 1 to adjust f 17 ,f 26 ,f 35 to the same size (640, 640, 64), and then splice the resized f 17 ,f 26 ,f 35 Into a whole, get the feature map f 172635 , f 172635 The size of f is (640, 640, 192), then let f 172635 Then, through a convolution layer with a 7x7 convolution kernel and a stride of 1, a multi-scale fusion feature map I is obtained, and the size of the multi-scale fusion feature map I is (640, 640, 3).
3. The method for tracking a vehicle's trajectory across cameras at night according to claim 1, characterized in that: Step (3) specifically includes the following steps in order: (3a) Normalizing the multi-scale fusion feature map I to obtain the normalized data I0; (3b) Constructing an MCAM-YOLO network model and processing the normalized data I0: using the YOLOv12 network as the basic network, the YOLOv12 network includes a backbone network, a neck network, and a detection head, and replacing the A2C2f module in the backbone network with an MCAC3k module; the MCAC3k module includes a convolution layer with a convolution kernel size of 1x1 and a stride of 1, a convolution layer with a convolution kernel size of 3x3 and a stride of 1, an average pooling layer, and a bias pooling layer; (3c) Output the detection results from the detection head.
4. The method for tracking a vehicle's trajectory across cameras at night according to claim 1, characterized in that: Step (4) specifically includes the following steps in order: (4a) Divide the detection box into high confidence detection boxes Bs according to the confidence level high With low confidence detection box Bs low Two categories, the high confidence detection box Bs high The image in the image is used for ReID feature extraction, and the extracted feature information is f i k ; Use the FastReID model with ResNeSt50 as the backbone to extract the location information of the vehicle in the detection frame and compare the location information with the high confidence detection frame Bs high Embed the appearance information in: Where, F i k is the appearance motion feature vector obtained by embedding the appearance and position of the current matching detection, and are the detection box parameters of the previous frame and this frame of the i-th trajectory, β1 and β2 are the weights of the appearance and position information embedding, respectively. It is the intersection-over-union ratio between the current detection box and the matching box at the previous moment: Where (x1, y1) is the bounding box The coordinates of the lower left corner, (x2, y2) is the bounding box The coordinates of the upper right corner; (x 1' ,y 1' ) is the bounding box The coordinates of the lower left corner, (x 2' ,y 2' ) is the bounding box The coordinates of the upper right corner, (x int1 ,y int1 ) is the bounding box With bounding box The coordinates of the lower left corner of the intersection area, (x int2 ,y int2 ) is the coordinate of the upper right corner of the intersection area of bounding box b1 and bounding box b2; The exponential moving average mechanism is used to update the matching trajectory appearance position information state e of the i-th trajectory at the k-th frame i k : Among them, α = 0.9 is the momentum term; Position information status in average trajectory appearance and F i k Match between them and measure the cosine similarity; (4b) Perform camera motion compensation and use OpenCV's global motion estimation technology to represent background motion. First, extract image key points, then use sparse optical flow to perform feature tracking based on translation-based local outlier suppression. First, use the random sampling consensus algorithm to calculate the affine transformation matrix. The predicted bounding box is then transformed from the coordinate system of the k-1 frame to the coordinate system of the next frame k using the affine transformation matrix. The translation part of the affine transformation matrix only affects the center position of the bounding box, while the other part affects all state vectors and noise matrices. Camera motion compensation is performed by the following equation: in, Represents the motion transformation matrix of the camera from the k-1th moment to the kth moment, M 2x2 represents the linear transformation matrix of the camera from time k-1 to time k, M is the scaling and rotation matrix; T 2x1 Represents the translation transformation matrix of the camera from the k-1th moment to the kth moment, is the state vector dimension transformation matrix, is the bounding box coordinate transformation matrix; Then combine camera motion compensation with Kalman filtering: Where, They are the predicted state vectors of the Kalman filter at time k before and after camera motion compensation, P k|k-1 , P′ k|k-1 are the prediction covariance matrices of the Kalman filter before and after correction, respectively; (4c) F i k and Combine and associate the first-level trajectory with the detection frame; the cost matrix used in the association process is C, which is combined with the appearance cosine matrix d embedded in the motion information cos and IoU distance matrix d iou ; d cos Elements is the appearance motion feature vector F in the i-th track i and the appearance motion feature vector F of the jth trajectory j The cosine distance between: Where, the IoU distance matrix d iou Elements in The IoU distance between the predicted bounding box i and the detected bounding box j of the trajectory is represented as IoU(b i ,b j ): Among them, b i is the coordinate information of the i-th predicted bounding box, b j is the coordinate information of the jth detection bounding box; Then, the minimum value of each element of the matrix is used as the final value of the cost matrix C to fuse the IoU information with the ReID information. The resulting cost matrix C of the fused IoU-ReID information is as follows: Among them, C i,j is the (i,j) element of the cost matrix C, is the IoU distance between the i-th predicted bounding box and the j-th detected bounding box of the track; is the appearance feature F embedded with motion information i and the newly detected appearance feature F embedded with motion information j The cosine distance between for Corrected distance; θ iou is the proximity threshold, set to 0.5; θ emb is the appearance threshold; (4d) According to C i,j The Hungarian algorithm is used for matching. The detected target is associated with the existing tracking trajectory. The set of successfully matched trajectories is T o1 ; During the matching process, the Static Sensitive Association (SSA) strategy is used. When the vehicle detection frame is detected to be stationary, the position and appearance state information of the corresponding track will be saved until the vehicle starts to activate it and participate in the matching process. (4e) Track the trajectory bidirectionally: first, forward tracking is performed, running the tracker from front to back on the video frame, generating a trajectory for each target based on the detected target and the trajectory information of the previous frame; then, backward tracking is performed, running the tracker forward from the last frame of the video, so that targets close to the camera are easier to detect and the trajectory is initialized; finally, the tracks are merged to obtain a complete trajectory; (4f) The low confidence detection box Bs low Perform secondary matching with the unmatched trajectories in step (4d). This matching is based on IoU. First, calculate the IoU matrix: calculate the prediction box updated by Kalman filter for each trajectory With Bs low The intersection-over-union ratio between the detection boxes in the , constitutes the matrix IoU, the matrix elements are: in, is the detection box predicted by trajectory i using Kalman filtering at time t+1, B j,t It's Bs low The j-th detection box in ; Then the Hungarian algorithm is used to match the intersection-over-union matrix as the cost matrix IoU. The set of successfully matched trajectories is T o2 ; (4g) Relink the trajectory: According to T o1 With T o2 The starting and ending positions of the trajectory in the , if the starting or ending coordinates of the trajectory fall within the area of the pre-calibrated abnormality, it is classified into the abnormal trajectory set T e , for the set T e Relink the trajectories in Use f i k Construct the cosine distance matrix C f , take the set T e The i-th trajectory T in ei and the jth trajectory T ej , C f Element C fi,j For these two trajectories T ei With T ej The cosine distance of the appearance feature: Then use the greedy algorithm to merge; (4h) T o1 With T o2 To manage and update the trajectory in , first update the state of the Kalman filter, and then update the appearance information in the trajectory.
5. The method for tracking a vehicle's trajectory across cameras at night according to claim 1, characterized in that: Step (5) specifically includes the following steps in order: (5a) Divide the road under different cameras into entrance and exit areas, where the camera set is N is the total number of cameras. The camera entrance and exit areas are defined and matched using the adjacency matrix A∈{0,1} N×N , for two cameras C ami with C amj ,for A ij =1 means C ami The exit area and C amj The entrance area is adjacent to the (5b) Using time information and entrance and exit matching information, filter the trajectories under different cameras; each camera C ami The trajectory collection in, Indicates camera C ami The trajectory of T i k Timestamp of entering the camera area, Indicates camera C ami The trajectory of T i k Timestamp of leaving the camera area, F i k Represents the appearance motion feature vector obtained by matching the appearance and position embedding of the detection, Indicates area information; take camera C ami Any trajectory T in i m With camera C amj Any trajectory in Among all the trajectories, keep two trajectories T that satisfy the following time and space consistency constraints i m and in, T is the trajectory i m Enter camera C ami Regional timestamp, For the trajectory Camera C amj Timestamp, Δt max is the maximum allowed time difference, F i m T is the trajectory i m The appearance motion feature vector, F i n T is the trajectory i n The appearance motion feature vector, θ feat is the feature similarity threshold; Get the trajectory set T that meets the entrance, exit and time constraints after filtering s '; (5c) for T s Each of the two different trajectories T m 'With T n ', using F i k , calculate the similarity matrix S, K is T s 'The number of trajectories in the The nth element in the mth row of the matrix S mn The formula is: Where, It represents the function of deep feature extraction using feature extraction network; (5d) Use the Hungarian algorithm to find the most fitting trajectory and save the result in the binary matching matrix X: Then use the Hungarian algorithm to solve the element X mn is the element of the binary matching matrix; when X mn When the value is 1, it means the trajectory T i m and trajectory Matching; the final matching results are saved as matching matrix X * ; (5e) According to the matched trajectory, correct the trajectory T' successfully matched in step (5d), and use the matched trajectory to correct T i k and Update the logo: Among them, G(x) is the global ID allocation function, is the matching matrix X * Elements, Indicates camera C amj The nth trajectory in T i m Indicates camera C ami The mth trajectory in For camera C amj The unified global trajectory ID of the nth trajectory in .
6. The method for tracking a vehicle's trajectory across cameras at night according to claim 3, characterized in that: In step (3b), the processing of the MCAC3k module includes the following steps: (3b1) Input I0 into the MCAM-YOLO network model to obtain the input I1 of the MCAC3k module. Then, a convolution layer with a convolution kernel size of 1x1 and a stride of 1 is used to adjust the number of channels to half of I1. Then, a convolution layer with a convolution kernel size of 3x3 and a stride of 1 is used to extract information to obtain the feature map I2. (3b2) Horizontal processing is performed on the feature map I2; the horizontal processing includes exchanging the channel dimension and the height dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3w ; (3b3) Processing the feature map I2 in the vertical direction; the vertical processing includes exchanging the channel dimension and the width dimension, performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling; obtaining the feature map I 3h ; (3b4) Processing the feature map I2 in the channel direction; the channel direction processing includes performing average pooling and deviation pooling, and performing weighted averaging on the results of average pooling and standard deviation pooling to obtain the feature map I 3c ; (3b5) for feature map I 3w , I 3h , I 3c Add and average to obtain the processed feature map I4, then add the residual of I4 and I2 to obtain the feature map I5; then let I5 pass through a convolution layer with a convolution kernel size of 1x1 and a stride of 1 to obtain the feature map I6, and finally, I6 is residually connected to I1 to output the feature map I7.
7. The method for tracking a vehicle's trajectory across cameras at night according to claim 4, characterized in that: In step (4g), the merging is performed using a greedy algorithm, and the process includes the following steps: (4g1) Arrange the fracture tracks according to the starting time; (4g2) Starting from the first trajectory, find the trajectory with the smallest cosine distance to its ReID feature among the remaining trajectories; (4g3) If the minimum distance is less than the set threshold, merge the two trajectories and update the trajectory set; otherwise, keep the current trajectory alone; (4g4) Repeat steps (4g1) to (4g3) until all trajectories are processed and a merged trajectory set is obtained.
8. An electronic device comprising: processor; as well as A memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the nighttime vehicle cross-camera trajectory tracking method according to any one of claims 1 to 7.
9. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to execute the method for tracking a vehicle trajectory across cameras at night according to any one of claims 1 to 7.
Citation Information
Patent Citations
Cross-space-time correlation method and device for night target trajectory
CN117495913A
Vehicle multi-target intelligent dynamic fusion tracking method
CN118608563A
Cited By
Vehicle speed detection method, electronic equipment, readable storage medium and program product
CN120668955A
Vehicle thermal imaging intelligent sensing method linked with aerial imaging system
CN121937446A
