Vehicle tracking and re-identification method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of vehicle tracking, and discloses a vehicle tracking and re-identification method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of monitoring images, and carrying out the preprocessing of the monitoring images; performing real-time target detection on the preprocessed monitoring image, identifying vehicles in the image, and generating an independent detection frame and a serial number for each identified vehicle; performing real-time tracking on the detected vehicle by using a multi-target tracking algorithm to generate a motion track of the vehicle; for the identified vehicles, using a deep convolutional neural network model to extract appearance features of the vehicles, and generating multi-dimensional feature vectors of a plurality of vehicles; calculating the similarity among the multi-dimensional feature vectors of the plurality of vehicles by using a similarity measurement method to obtain similarity values of the plurality of vehicles; the multiple vehicles with the similarity values higher than a preset threshold value are marked as the same vehicle, and vehicle re-identification is achieved; the accuracy of vehicle tracking and the accuracy of vehicle re-identification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle tracking, and in particular to a vehicle tracking and re-identification method, device, electronic equipment and storage medium. Background Art

[0002] With the continuous expansion of urban scale and the sharp increase in the number of vehicles, urban traffic has become more and more complex and congested. There are many types of vehicles in urban traffic, including cars, buses, taxis, trucks, etc. These vehicles have different driving patterns and behavior characteristics in different sections and time periods. How to effectively track and identify these vehicles has become a major problem facing traffic management departments. Existing traffic management methods mainly rely on manual monitoring and fixed cameras, which are difficult to fully cover and monitor in real time, are inefficient and easy to miss.

[0003] With the rapid development of deep learning and computer vision technology, video-based vehicle tracking and re-identification technology has emerged. By deploying high-definition cameras at key locations in the city, combined with advanced target detection and tracking algorithms, real-time monitoring and accurate identification of vehicles can be achieved. These technologies can not only be used for traffic flow analysis and management, but also effectively identify and combat traffic violations such as running red lights, speeding, illegal parking, etc.

[0004] However, vehicle tracking and re-identification technology still faces many challenges in practical applications. For example, different types of vehicles have very different appearances, environmental factors such as weather and lighting have a significant impact on image quality, and vehicles may be blocked and change during driving, which results in low accuracy in vehicle tracking and identification. Summary of the invention

[0005] In view of the deficiencies in the prior art, the object of the present invention is to provide a vehicle tracking and re-identification method, device, electronic device and storage medium.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A vehicle tracking and re-identification method comprises the steps of:

[0008] Acquire multiple surveillance images, and pre-process the surveillance images;

[0009] Perform real-time target detection on the preprocessed surveillance image, identify the vehicles in the image, and generate an independent detection frame and number for each identified vehicle;

[0010] Use multi-target tracking algorithms to track detected vehicles in real time and generate vehicle motion trajectories;

[0011] For the identified vehicles, a deep convolutional neural network model is used to extract the appearance features of the vehicles and generate multi-dimensional feature vectors of multiple vehicles;

[0012] Using a similarity measurement method to calculate the similarity between the multi-dimensional feature vectors of multiple vehicles, and obtaining similarity values ​​of the multiple vehicles;

[0013] Multiple vehicles with similarity values ​​higher than a preset threshold are marked as the same vehicle to achieve vehicle re-identification.

[0014] In the present invention, preferably, the target detection is performed using a target detection model, the target detection model includes an input layer, a convolutional layer, a residual layer, an upsampling layer, a routing layer and an output layer, and the target detection model identifies and obtains a detection frame of the vehicle.

[0015] In the present invention, preferably, the convolutional layer sequentially includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer and a third pooling layer.

[0016] In the present invention, preferably, the generating of the motion trajectory of the vehicle specifically comprises the steps of:

[0017] Initialize a new track for each vehicle detected in the current frame, record its position and features, and get the detection result;

[0018] The Hungarian algorithm is used to calculate the similarity between the detection results of multiple vehicles and the existing motion trajectory information to obtain the data association results of multiple vehicles;

[0019] According to the data association results of the multiple vehicles, the track states of the multiple vehicles are updated.

[0020] In the present invention, preferably, updating the trajectory states of multiple vehicles includes:

[0021] For newly detected vehicles, new tracks are created; for tracks that fail to match detection results, they are marked as lost and deleted after a number of frames.

[0022] In the present invention, preferably, the trajectory state includes positions, speeds and appearance features of multiple vehicles.

[0023] In the present invention, preferably, the multidimensional feature vector includes:

[0024] The color features of the vehicle are extracted through the color histogram and color moment methods;

[0025] Extracting vehicle shape features through edge detection and region-based shape descriptors;

[0026] License plate recognition technology is used to extract license plate information as auxiliary features.

[0027] A vehicle tracking and re-identification device, comprising:

[0028] A real-time monitoring image module is used to obtain multiple monitoring images and pre-process the monitoring images, including image denoising and illumination correction, to ensure that the quality of the pre-processed images meets the standards for tracking and re-identification;

[0029] The target detection module is used to perform real-time target detection on the preprocessed image using a model trained based on a deep learning algorithm, identify vehicles in the image, and generate an independent detection frame and number for each identified vehicle;

[0030] The trajectory generation module is used to track the detected vehicles in real time using a multi-target tracking algorithm and generate the vehicle's motion trajectory;

[0031] A feature extraction module, for extracting appearance features of multiple vehicles detected in multiple surveillance images using a deep convolutional neural network, and combining multiple feature extraction methods to generate multi-dimensional feature vectors of the multiple vehicles;

[0032] A measurement module, used for calculating the similarity between the multidimensional feature vectors of the plurality of vehicles using a similarity measurement method to obtain similarity values ​​of the plurality of vehicles;

[0033] The identification module is used to mark the multiple vehicles as the same vehicle if the similarity values ​​of the multiple vehicles are higher than a preset threshold value, so as to achieve vehicle re-identification.

[0034] An electronic device, comprising:

[0035] at least one processor; and

[0036] a memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the vehicle tracking and re-identification method.

[0038] A storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the vehicle tracking and re-identification method as described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) Improving the accuracy of vehicle tracking and re-identification: The present invention uses a multi-target tracking algorithm based on the Hungarian algorithm to calculate the similarity between the detection results of multiple vehicles and the existing trajectory information. Based on the similarity, the data correlation of multiple vehicles can be better matched, thereby improving the accuracy of vehicle tracking; in addition, by extracting multiple features of the vehicle including color features, shape features and auxiliary features, the accuracy of vehicle re-identification is improved.

[0041] (2) Enhanced environmental adaptability: The present invention significantly improves the adaptability of the vehicle tracking and re-identification system in different monitoring environments by performing image preprocessing including image denoising and illumination correction, as well as extracting multiple features of the vehicle. These technologies enable the system to effectively cope with different lighting conditions, weather changes, and background complexity, thereby ensuring high accuracy and reliability in various environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a schematic diagram of a vehicle tracking and re-identification method.

[0043] Figure 2 This is a schematic flow chart of step S3 of the invention.

[0044] Figure 3 It is a schematic flow chart of step S4 described in the invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0047] Please also see Figures 1 to 3 A preferred embodiment of the present invention provides a vehicle tracking and re-identification method, comprising the steps of:

[0048] S1. Acquire multiple surveillance images and pre-process the surveillance images;

[0049] S2, performing real-time target detection on the preprocessed surveillance image, identifying the vehicles in the image, and generating an independent detection frame and number for each identified vehicle;

[0050] S3, using a multi-target tracking algorithm to track the detected vehicle in real time and generate the vehicle's motion trajectory;

[0051] S4. For the identified vehicles, use a deep convolutional neural network model to extract the appearance features of the vehicles and generate multi-dimensional feature vectors of multiple vehicles;

[0052] S5. Calculate the similarity between the multidimensional feature vectors of the multiple vehicles using a similarity measurement method to obtain similarity values ​​of the multiple vehicles;

[0053] S6. Mark multiple vehicles whose similarity values ​​are higher than a preset threshold as the same vehicle to achieve vehicle re-identification.

[0054] Specifically, in step S1, multiple monitoring images are obtained, and the monitoring images are preprocessed. The preprocessing includes denoising the monitoring images and correcting the illumination to ensure that the quality of the preprocessed images meets the standards for tracking and re-identification. The acquisition of real-time monitoring images includes equipment configuration and real-time data acquisition. A high-definition camera is installed in a designated monitoring area to capture clear images, and the collected image data is stored in a designated storage space. Then, the image is preprocessed. First, the image clarity value is calculated using an image clarity algorithm (Laplacian algorithm), and images with clarity values ​​below a set threshold are deleted. Then, the environmental noise is filtered, and the dynamic background (such as moving values, passing vehicles) and the static foreground (such as parked non-motor vehicles) are separated by a dynamic background filtering method combined with a Gaussian mixture model. Automatic exposure correction is performed by an illumination adjustment method, and the contrast and brightness of the image are adjusted to keep the image clear under different illuminations, eliminating environmental noise caused by weather, illumination changes, moving objects, etc. Finally, the image is standardized, all images are uniformly converted to a set format, and adjusted to a uniform size to ensure that the image data format and size are unified to adapt to subsequent deep learning model processing.

[0055] In this embodiment, the pre-processed image data can be temporarily stored in a local or cloud database to facilitate subsequent processing and analysis.

[0056] Specifically, in step S2, real-time target detection is performed on the preprocessed monitoring image, and a target detection model is used for target detection. The target detection model includes an input layer, a convolution layer, a residual layer, an upsampling layer, a routing layer and an output layer, and the target detection model identifies and obtains the detection frame of the vehicle. The input layer is used to receive the real-time image and cut it into an image with a resolution of 256x256 pixels; the convolution layer includes the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the third convolution layer and the third pooling layer in sequence, and each convolution layer is provided with a normalization and ReLU activation function to extract features of different scales; each residual layer is used to deepen the network without losing information, helping to train a deeper network structure; the upsampling layer is used to increase the size of the feature map in order to capture more detailed information; the routing layer merges the outputs of different layers to increase the flexibility and complexity of the network; the output layer uses a convolution operation to output a three-dimensional feature, and each cell contains information (position, size, confidence and classification probability) of multiple bounding boxes.

[0057] In this embodiment, the filter size of the first convolution layer of the convolution layer is 3x3, the number is 64, the step size is 1, and the window size of the first pooling layer is 2x2, the step size is 2; the filter size of the second convolution layer is 3x3, the number is 128, the step size is 1; the window size of the second pooling layer is 2x2, the step size is 2; the filter size of the third convolution layer is 3x3, the number is 256, the step size is 1, and the window size of the third pooling layer is 2x2, the step size is 2. The convolution layer using this structure can achieve a better feature extraction effect. After the third convolution layer, a residual layer is added to superimpose the input of the third convolution layer with the output of the third convolution layer, thereby reducing the information loss and gradient disappearance problems in the training process and ensuring the effective transmission of information. After the third pooling layer, an upsampling layer is set, and a bilinear interpolation method is used to increase the size of the feature map from 64x64 pixels to 128x128 pixels, so that some spatial details lost in the pooling process can be restored. The routing layer merges the outputs of different layers to increase the flexibility and complexity of the network. The outputs of the second and third convolutional layers are merged so that the network can simultaneously utilize shallow detail information and deep advanced features, effectively integrating features at different levels. At the end of the target detection model, a convolutional layer is used as the output layer, and multiple filters are set. Each filter is responsible for detecting a specific object in the image. The output of the output layer is a three-dimensional feature map, in which each cell contains information about the detected object, such as position, size, confidence, and classification probability. With such a setting, the target detection model can effectively process the input image, extract and fuse features layer by layer, and finally generate accurate target detection results.

[0058] In this embodiment, the target detection model needs to be obtained through preliminary training. The collected and labeled vehicle images are input into the original target detection model, and the cross entropy loss function is used to optimize the classification error. At the same time, the mean square error loss function is used to optimize the prediction of the bounding box. A gradually decreasing learning rate, as well as techniques such as weight decay and momentum optimization, are used to ensure stable learning of the model. Multiple iterations (Epochs) are performed until the accuracy and loss of the model are stabilized within an acceptable range. The target detection model is then verified using an image set that has not participated in the training to ensure that the target detection model has good generalization ability and can effectively perform vehicle detection tasks in monitoring images.

[0059] In this embodiment, the target detection model is used to identify the vehicles in the image, including the location, size and possible types of the vehicles, such as motor vehicles, electric vehicles, etc., and an independent detection frame and corresponding number are generated for each identified vehicle. The detection frame marks the location and boundary of the vehicle. The detection frame contains: the coordinates (x, y) and size (width and height) of the frame, the confidence score for whether the detection frame contains non-motor vehicles, and the category to which the identified motor vehicle may belong and its corresponding probability data. The corresponding number is used to number the identified vehicle type, and the number is used to distinguish different vehicles. For example, in a surveillance video of a certain city street, the target detection model identifies a small passenger car parked on the sidewalk. The target detection model first identifies the shape and features of the vehicle in the image through multi-layer convolution and feature extraction processes. Then, based on these features, the target detection model generates a detection frame that accurately covers the position of the small passenger car. The detection frame contains the exact location of the small passenger car, the size of the frame, and the recognition result with high confidence, and assigns it a label value. Through this step, the system can accurately identify and locate vehicles from the surveillance image in real time, providing key data for subsequent analysis.

[0060] Specifically, in step S3, a multi-target tracking algorithm is used to track the detected vehicle in real time to generate a motion trajectory of the vehicle, which specifically also includes steps S301-S304.

[0061] In step S301, the multi-target tracking algorithm adopts the DeepSORT (Deep Simple Online and Realtime Tracking) model, combining deep learning appearance feature extraction with traditional Kalman filtering and Hungarian algorithm to achieve efficient multi-target tracking. First, the vehicle detection frame generated in step S2 is subjected to a practical deep convolutional neural network (ResNet50) to extract the appearance feature vector of each detected vehicle, and a 128-dimensional appearance feature vector of each vehicle is obtained.

[0062] The vehicle state vector x is obtained according to the following formula (1.1), which needs to include information such as the vehicle's position, speed, and shape:

[0063]

[0064] in:

[0065] x, y are the center position coordinates of the vehicle;

[0066] a is the aspect ratio of the vehicle;

[0067] h is the height of the vehicle;

[0068] is the velocity of the position;

[0069] is the ratio of aspect ratio and rate of change of height;

[0070] Finally, the Kalman filter is used to predict the state of the current time step using the state estimation of the previous time step and the state transition model. The state transition model describes the evolution of the vehicle state from one time step to the next time step.

[0071] Formula (1.2) state transfer equation:

[0072] x k =Fx k-1 +w k-1 (1.2)

[0073] in:

[0074] x k is the state vector at time k

[0075] F is the state transfer matrix

[0076] w k-1 is the process noise, which is assumed to be normally distributed with a mean of zero;

[0077] In the prediction phase, the Kalman filter uses the state estimate from the previous time step and the state transition model to predict the state at the current time step.

[0078] According to formula (1.3), the state vector is predicted:

[0079]

[0080] in is the predicted state at time k.

[0081] According to formula (1.4), the covariance matrix is ​​predicted:

[0082]

[0083] in is the prediction covariance matrix at time k, P k-1 is the state covariance matrix of the previous time step, and Q is the process noise covariance matrix.

[0084] Step S302, initialize a new track for each vehicle detected in the current frame, record its position and features, and obtain the detection result.

[0085] Step S303: using the Hungarian algorithm, similarities are calculated based on the detection results of multiple vehicles and the existing trajectory information to obtain data association results of the multiple vehicles.

[0086] For each appearance feature vector of the detected vehicle extracted in step S301, the cosine similarity is calculated using formula (1.5):

[0087]

[0088] where f i and f j are the feature vectors of detection and trajectory respectively.

[0089] Then the similarities between all detections and all trajectories are calculated to obtain the association matrix.

[0090] Finally, the Hungarian algorithm is used to find the best match in the association matrix so that the total similarity is maximized or the total cost is minimized.

[0091] Step S304, based on the data association results of the multiple vehicles, the trajectory states of the multiple vehicles are updated. At each time step, after the Kalman filter completes the prediction and update, each trajectory state is updated, and the trajectory state includes the position, speed and shape information of the vehicle. For newly detected vehicles, a new trajectory is initialized; for trajectories that have not been detected for a long time, they are marked as terminated and deleted from the active trajectory list.

[0092] Specifically, in step S4, for the identified vehicles, a deep convolutional neural network model is used to extract the appearance features of the vehicles and generate multi-dimensional feature vectors of multiple vehicles. First, the appearance features are extracted from the vehicle image using the DenseNet network structure to obtain the appearance feature vector of each vehicle; at the same time, the features of each vehicle are extracted by combining multiple feature extraction methods, including: extracting the color features of the vehicle through color histogram and color moment method; extracting the shape features of the vehicle through edge detection and region-based shape descriptors; and extracting the license plate information as an auxiliary feature using license plate recognition technology. DenseNet uses densely connected blocks (Dense Blocks) so that each layer directly receives the feature maps of all previous layers as input, thereby improving the efficiency of feature transfer and reducing the gradient vanishing problem.

[0093] Specifically, in step S5, a similarity measurement method is used to calculate the similarity between the multidimensional feature vectors of multiple vehicles to obtain the similarity values ​​of multiple vehicles. In the multi-target tracking and re-identification task, by calculating the similarity between multiple feature vectors of multiple vehicles, it can be determined whether they are the same vehicle. The cosine similarity is still used to calculate the cosine value of the angle between two feature vectors. The larger the value, the more similar it is.

[0094] Specifically, in step S6, multiple vehicles with similarity values ​​higher than a preset threshold are marked as the same vehicle to achieve vehicle re-identification. By calculating the similarity values ​​of multiple vehicles and comparing them with the preset threshold, it can be determined whether these vehicles are the same vehicle. A similarity threshold is set to determine whether the vehicle is the same vehicle. The similarity threshold can be adjusted according to the specific application scenario, such as 0.75. According to the similarity matrix and the preset threshold, vehicles with similarity values ​​higher than the threshold are marked as the same vehicle, and the vehicle number is displayed on the image to indicate that it is the same vehicle. Through the above steps, the detected vehicle features can be comprehensively analyzed, and the marking of the same vehicle can be achieved by calculating the similarity matrix and comparing it with the preset threshold. This can effectively improve the accuracy of vehicle re-identification, especially in complex urban traffic environments, and help improve the performance and robustness of multi-target tracking systems.

[0095] In some other preferred embodiments of the present invention, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the above embodiment.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0097] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A vehicle tracking and re-identification method, characterized in that: Includes steps: Acquire multiple surveillance images, and pre-process the surveillance images; Perform real-time target detection on the preprocessed surveillance image, identify the vehicles in the image, and generate an independent detection frame and number for each identified vehicle; Use multi-target tracking algorithms to track detected vehicles in real time and generate vehicle motion trajectories; For the identified vehicles, a deep convolutional neural network model is used to extract the appearance features of the vehicles and generate multi-dimensional feature vectors of multiple vehicles; Using a similarity measurement method to calculate the similarity between the multi-dimensional feature vectors of multiple vehicles, and obtaining similarity values ​​of the multiple vehicles; Multiple vehicles with similarity values ​​higher than a preset threshold are marked as the same vehicle to achieve vehicle re-identification. Generating the motion trajectory of the vehicle specifically includes the following steps: Initialize a new track for each vehicle detected in the current frame, record its position and features, and get the detection result; The Hungarian algorithm is used to calculate the similarity between the detection results of multiple vehicles and the existing motion trajectory information to obtain the data association results of multiple vehicles; According to the data association results of the multiple vehicles, the track states of the multiple vehicles are updated.

2. A vehicle tracking and re-identification method according to claim 1, characterized in that: The real-time target detection adopts a target detection model, which includes an input layer, a convolution layer, a residual layer, an upsampling layer, a routing layer and an output layer. The target detection model recognizes and obtains a detection frame of the vehicle.

3. A vehicle tracking and re-identification method according to claim 2, characterized in that: The convolutional layers sequentially include a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer and a third pooling layer.

4. A vehicle tracking and re-identification method according to claim 3, characterized in that: The updating of the trajectory states of the plurality of vehicles comprises: For newly detected vehicles, new tracks are created; for tracks that fail to match detection results, they are marked as lost and deleted after a number of frames.

5. A vehicle tracking and re-identification method according to claim 4, characterized in that: The trajectory state includes positions, speeds, and appearance features of multiple vehicles.

6. A vehicle tracking and re-identification method according to claim 1, characterized in that: The multidimensional feature vector includes: The color features of the vehicle are extracted through the color histogram and color moment methods; Extracting vehicle shape features through edge detection and region-based shape descriptors; License plate recognition technology is used to extract license plate information as auxiliary features.

7. A vehicle tracking and re-identification device, characterized in that: include: A real-time monitoring image module is used to obtain multiple monitoring images and pre-process the monitoring images, including image denoising and illumination correction, to ensure that the quality of the pre-processed images meets the standards for tracking and re-identification; The target detection module is used to perform real-time target detection on the preprocessed image using a model trained based on a deep learning algorithm, identify vehicles in the image, and generate an independent detection frame and number for each identified vehicle; The trajectory generation module is used to track the detected vehicles in real time using a multi-target tracking algorithm and generate the vehicle's motion trajectory; A feature extraction module, for extracting appearance features of multiple vehicles detected in multiple surveillance images using a deep convolutional neural network, and combining multiple feature extraction methods to generate multi-dimensional feature vectors of the multiple vehicles; A measurement module, used for calculating the similarity between the multidimensional feature vectors of the plurality of vehicles using a similarity measurement method to obtain similarity values ​​of the plurality of vehicles; The identification module is used to mark the multiple vehicles as the same vehicle if the similarity values ​​of the multiple vehicles are higher than a preset threshold value, so as to achieve vehicle re-identification.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vehicle tracking and re-identification method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of a vehicle tracking and re-identification method as described in any one of claims 1 to 6.

Images