Deep learning-based camera-crossing vehicle target tracking method, system and device in expressway tunnel, and medium

Abstract

The invention relates to the technical field of artificial intelligence image processing, and particularly discloses a camera-crossing vehicle target tracking method, system and device in an expressway tunnel based on deep learning and a medium, and the method comprises the steps: image collection, image preprocessing, detection and recognition of the preprocessed image through a trained YOLOv5 model, and detection and recognition of the camera-crossing vehicle target in the expressway tunnel through the trained YOLOv5 model. A vehicle in an image is identified, a detection frame is generated for the vehicle, and then the type and color of the vehicle, vehicle re-identification Re-id, single-camera vehicle target tracking and cross-camera vehicle target tracking are marked. According to the method, image preprocessing model adaptive image enhancement is combined with YOLOv5, so that the type and color of the vehicle can be accurately detected under foggy days and low-illumination conditions, the feature vector after Re-id is added on the basis of the ByteTrack tracking algorithm to serve as track matching of a low-confidence detection frame, and compared with a traditional ByteTrack tracking algorithm, the method has the advantages that the method is simple and convenient to operate, and the detection efficiency is improved. Through the feature vector output by the ResNet101-IBN model, the static feature of the target can be obtained in a more detailed manner, and the accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing technology, and in particular to a method, system, device and medium for tracking vehicle targets across cameras in a highway tunnel based on deep learning. Background Art

[0002] In the modern transportation system, highway tunnels are key traffic nodes that carry huge traffic volume. Traffic safety management in tunnels faces severe challenges, especially in terms of accident prevention, emergency response, and traffic flow optimization. However, poor lighting conditions, closed spaces, and limited vision in tunnels make vehicle target detection and monitoring particularly difficult. At present, vehicle target detection in tunnels mainly relies on traditional video surveillance systems and human eye-to-video methods, which have problems such as poor environmental adaptability, insufficient real-time performance, difficulty in cross-camera tracking, and limited data processing capabilities.

[0003] The lighting conditions in highway tunnels are significantly different from those on open roads, especially at the entrance and exit of the tunnel. Due to the drastic changes in external light, the images captured by the camera may be overexposed or underexposed. Secondly, the occlusion between vehicles in the tunnel is a common problem, especially when a large vehicle briefly blocks a small vehicle, which can easily cause the ID of the small vehicle to switch. In order to achieve comprehensive monitoring of vehicle targets in highway tunnels, multiple cameras are usually deployed. These cameras need to be precisely synchronized to ensure data consistency, especially when the field of view of one camera is transferred to the field of view of another camera, it cannot be recognized as the same target vehicle, resulting in the failure of cross-camera tracking of the vehicle.

[0004] Therefore, an innovative and practical cross-camera vehicle target tracking method is urgently needed to solve the above problems and shortcomings of existing technologies. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a method, system, device and medium for cross-camera vehicle target tracking in a highway tunnel based on deep learning. By collecting video images under different environmental conditions in the highway tunnel and making a data set, the data set is used to train an image preprocessing model and a target detection model, and feature extraction is performed according to the results of target detection to realize single-camera target tracking. Hierarchical clustering and constraints are combined to realize cross-camera vehicle target tracking, thereby solving the problems mentioned in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for tracking a vehicle target across cameras in a highway tunnel based on deep learning, comprising the following steps:

[0007] S1. Image acquisition: Install multiple cameras at different positions in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color to form a data set;

[0008] S2, image preprocessing: construct an image preprocessing model, and preprocess the video image through the image preprocessing model;

[0009] S3, vehicle target detection: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle;

[0010] S4, Vehicle Re-ID: Use the trained Resnet101-IBN model to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle; then use the trained SVM classifier to classify the feature vectors to obtain more detailed classification results of vehicle type and color;

[0011] S5, single camera vehicle target tracking: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained;

[0012] S6. Cross-camera vehicle target tracking: Match the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

[0013] Preferably, the image preprocessing model includes a lightweight CNN-based parameter predictor ShuffleNet V2 and a differentiable image processing module DIP; the lightweight CNN-based parameter predictor ShuffleNet V2 consists of a ShuffleNetV2 downsampling module, three ShuffleNetV2 feature extraction networks and two fully connected layers; the differentiable image processing module DIP consists of six differentiable filters for dehazing, white balance, gamma, contrast, hue and sharpening.

[0014] Preferably, in step S2, it specifically includes: downsampling the video image in the data set through the ShuffleNetV2 downsampling module, then extracting features through the ShuffleNetV2 feature extraction network and inputting them into the differentiable image processing module DIP, performing dehazing, white balance, gamma, contrast, hue and sharpening processing on the image to obtain the preprocessed image.

[0015] Preferably, the downsampling process includes: copying the input image into two copies; performing 3x3, stride=2, and 1x1 convolution on one copy; performing 1x1, 3x3, stride=2, and 1x1 convolution on the other copy; and finally splicing the outputs of the two branches in terms of the number of channels to complete the downsampling;

[0016] The feature extraction process includes: dividing the downsampled channel into two parts; performing 3x3, stride=2, and 1x1 convolution on one part, and not processing the other part; performing channel shuffling processing on the outputs of the two parts to complete feature extraction.

[0017] Preferably, in step S5, the following is specifically included:

[0018] S501, detecting and identifying detection frames with different confidence levels based on the YOLOv5 model;

[0019] S502, predict all active and inactive trajectories at frame 1 to generate a prediction frame, and perform an IOU calculation on the trajectory prediction frame and the high confidence detection frame at frame 2;

[0020] S503, 0.5 is used as the IOU threshold. When it is greater than or equal to 0.5, it is considered a successful match. At this time, the target appears at both frame 1 and frame 2 and is identified as the same target. The trajectory of the target is updated.

[0021] S504: The high-confidence target box that failed to match is identified as a new track, that is, it appears in the video for the first time. At this time, the new track is identified as an inactive state and needs to be matched three or more times in a row to become active.

[0022] S505, the unmatched trajectory will be matched with the low confidence detection frame of frame2, and the matching rule is the feature vector obtained by Re-id, and the matching is performed by calculating the cosine similarity of the feature vector, where the threshold of the cosine similarity is set to 0.5;

[0023] S506, updating the successfully matched target trajectory;

[0024] S507, the low confidence target box that fails to match will be discarded;

[0025] S508, the trajectories that failed to match include the activated state and the inactivated state, and the trajectories in the two states are further judged;

[0026] S509: If the active track has been mismatched 30 times, the track is deleted, indicating that the target frame has left the camera monitoring range;

[0027] S510: If the secondary matching of an inactive track fails, the track is directly deleted.

[0028] Preferably, in step S6, the following is specifically included:

[0029] S601, each target each multi-camera track T i All of them are composed of multiple single camera trajectories τ i The similarity of the trajectory under each camera is calculated by the cosine similarity between the mean features. The similarity of the trajectory of the target under each camera is calculated by the following formula:

[0030]

[0031] S602, using time as a constraint condition to achieve trajectory matching under different cameras, setting the time constraint between cameras to the conversion time between adjacent cameras τ1 and τ2;

[0032] S603, clustering the trajectories of the same target under different cameras by similarity between trajectories and constraints between cameras, selecting an average connection method to cluster the trajectories, and realizing tracking of the same target under different cameras, wherein the average connection clustering method is expressed as follows:

[0033]

[0034] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: a cross-camera vehicle target tracking system in a highway tunnel based on deep learning, comprising the following modules:

[0035] Image acquisition module: Install multiple cameras at different locations in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color to form a data set;

[0036] Image preprocessing module: build an image preprocessing model and preprocess the video image through the image preprocessing model;

[0037] Vehicle target detection module: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle;

[0038] Vehicle Re-ID module: The trained Resnet101-IBN model is used to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle. The trained SVM classifier is then used to classify the feature vectors to obtain more detailed classification results of vehicle type and color.

[0039] Single-camera vehicle target tracking module: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained;

[0040] Cross-camera vehicle target tracking module: Matches trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

[0041] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: an electronic device, the electronic device comprising: a processor; and a memory for storing one or more programs;

[0042] When the one or more programs are executed by the processor, the processor executes the deep learning-based cross-camera vehicle target tracking method in a highway tunnel.

[0043] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for cross-camera vehicle target tracking in a highway tunnel based on deep learning is implemented.

[0044] The beneficial effects of the present invention are:

[0045] 1) The present invention combines image preprocessing model adaptive image enhancement with YOLOv5, so that the type and color of the vehicle can be accurately detected in foggy and low-light conditions with good accuracy;

[0046] 2) When tracking a target, even if the target is blocked by other targets, the target can still match the ID when it first appears when it appears again. Different from the general ByteTrack tracking algorithm, the present invention uses a feature extraction network instead of a simple intersection-and-union ratio for trajectory matching, and pays more attention to the static properties of the target that are not easy to change, thereby improving the fault tolerance rate during target tracking, and the fault tolerance rate is high;

[0047] 3) The present invention is carried out in a special environment and has better processing and effects on low-quality images. Therefore, it can also be applied in environments of other qualities, which improves the adaptability and expansibility of the system and has high flexibility. Moreover, from image acquisition to cross-camera tracking, the whole process is automated without manual intervention, which improves the practicality and convenience of the system and has a high degree of automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the steps of a method for tracking a vehicle target across cameras in a highway tunnel based on deep learning according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the ShuffleNetV2 downsampling process according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the huffleNetV2 feature extraction process according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of vehicle re-identification Re-id features according to an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the improved ByteTrack target tracking process according to an embodiment of the present invention;

[0053] Figure 6 It is a pseudo code diagram of an algorithm based on hierarchical clustering according to an embodiment of the present invention;

[0054] Figure 7 This is a pseudo code diagram of a camera compatibility test function according to an embodiment of the present invention;

[0055] Figure 8 It is a schematic diagram of a module of a cross-camera vehicle target tracking system in a highway tunnel based on deep learning in an embodiment of the present invention;

[0056] Fig. 9 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention;

[0057] In the figure, 110-image acquisition module; 120-image preprocessing module; 130-vehicle target detection module; 140-vehicle re-identification Re-id module; 150-single-camera vehicle target tracking module; 160-cross-camera vehicle target tracking module; 210-processor; 220-storage. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1-Figure 8 ,The present invention provides a technical solution: a method for tracking vehicle targets across cameras in a highway tunnel based on deep learning, such as Figure 1 As shown, the following steps are included:

[0060] S1. Image acquisition: Install multiple cameras at different positions in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type (car, bus, truck) and color to form a data set.

[0061] S2. Image preprocessing: Build an image preprocessing model and preprocess the video image through the image preprocessing model.

[0062] The current method for detecting vehicle targets in highway tunnels usually labels the original images in the highway tunnels and directly trains the target detection model with the labeled data set. This method is usually suitable for situations where the tunnel environment is good. For some blurry cameras and foggy conditions, the results of this method are not ideal. The present invention proposes ShuffleNetV2 to learn an appropriate differentiable image processing module (DIP) in a weakly supervised manner to enhance the image for target detection.

[0063] In the scene of highway tunnel, visibility is poor due to the interference of weather and lighting specific information. To address this challenge, an image adaptive detection framework (i.e., image preprocessing model) is proposed to reveal more potential information by removing weather and lighting specific information. The image preprocessing model includes a lightweight CNN-based parameter predictor ShuffleNetV2 and a differentiable image processing module DIP; the lightweight CNN-based parameter predictor ShuffleNet V2 consists of a ShuffleNet V2 downsampling module, three ShuffleNet V2 feature extraction networks, and two fully connected layers, and the last fully connected layer outputs the hyperparameters of the DIP module. The output channels of these three convolutional layers are 32, 32, and 32 respectively; the differentiable image processing module DIP consists of six differentiable filters for dehazing, white balance, gamma, contrast, hue, and sharpening. An end-to-end mixed data training scheme is adopted so that the lightweight CNN can learn an appropriate DIP in a weakly supervised manner to enhance the image and use it for object detection.

[0064] The video images in the dataset are downsampled through the ShuffleNetV2 downsampling module, and then feature extracted through the ShuffleNetV2 feature extraction network and input into the differentiable image processing module DIP to perform dehazing, white balance, gamma, contrast, hue and sharpening on the image to obtain the preprocessed image.

[0065] like Figure 2 As shown, the downsampling process includes: copying the input image into two; performing 3x3 (stride=2) and 1x1 convolution on one of the copies; performing 1x1, 3x3 (stride=2), and 1x1 convolution on the other copy; finally, concatenating the outputs of the two branches in terms of the number of channels, so that the spatial size of the feature map is halved, but the number of channels is doubled, thereby achieving efficient feature map size reduction while maintaining rich features and completing downsampling.

[0066] like Figure 3 As shown, the feature extraction process includes: dividing the downsampled channel into two parts; performing 3x3 (stride=2) and 1x1 convolution on one part, and not processing the other part; performing channel shuffling processing on the outputs of the two parts, rearranging the channels, enhancing the information flow between different channels, and completing feature extraction.

[0067] White balance, gamma, contrast, and hue can be expressed as pixel-level filters that take the input pixel value P i =(r i , g i , b i) is mapped to an output pixel value P0 = (r0, g0, b0), where (r, g, b) represent the values ​​of the three color channels of red, green and blue, respectively. The mapping functions of the four pixel-level filters are listed in Table 1, where the second column lists the parameters that need to be optimized. White balance and gamma are simple multiplication and power transformations. The differentiable contrast filter involves an input parameter that is used to set the linear interpolation between the original image and the fully enhanced image.

[0068] Pixel-level filter parameter prediction process:

[0069] Image white balance adjustment, optimization parameters: W r , W g , W b ; Mapping function:

[0070] P0=(W r r i , W g g i , W b b i ) (1)

[0071] Image gamma adjustment, optimization parameters: G (gamma value); mapping function:

[0072] P0=P i G (2)

[0073] Image contrast adjustment, optimization parameters: α (contrast value); mapping function:

[0074] P0=α·E n (P i )+(1-α)·P i (3.1)

[0075] The mapping function En ( P i )as follows:

[0076] Lum(P i )=0.27r i +0.67g i +0.06b i (3.2)

[0077]

[0078] Image tone adjustment, optimization parameters: t i (hue value); mapping function:

[0079]

[0080] The hue filter is designed as a monotonic piecewise function. The hue filter is learned using L parameters, denoted as {t0, t1, ..., t L-1}. The tone curve is expressed as (k / L,T k / T L ),in The mapping function is expressed by a differentiable parameter, which makes the function relative to the input image and the parameters {t0, t1, ..., t L-1} can be differentiated as follows:

[0081]

[0082] Sharpening filter parameter prediction process:

[0083] I(x) is the input image, Gau(I(x) represents a Gaussian filter, and λ is a positive scaling factor. This sharpening operation is differentiable with respect to x and λ. The degree of sharpening can be adjusted by optimizing λ to improve object detection performance.

[0084] F(x, λ)=I(x)+λ(I(x)-Gau(I(x)))(5)

[0085] Dehaze filter parameter prediction process:

[0086] Based on the atmospheric scattering model, the formation of foggy images can be formulated as follows:

[0087] I(x)=J(x)t(x)+A(1-t(x))(6)

[0088] Where I(x) is the foggy image, J(x) represents the scene radiance (clear image), A is the global atmospheric light, and t(x) is the medium transmission diagram, which is defined as:

[0089] t(x)=e -β d(x) (7)

[0090] Where β represents the scattering coefficient of the atmosphere and d(x) is the scene depth.

[0091] In order to restore the clean image J(x), the key is to obtain the atmospheric light A and the transmission map t(x). To this end, first calculate the dark channel map I(x) of the haze image and select the brightest 1000 pixels. Then, estimate A by averaging the pixels of these 1000 pixels at the corresponding positions of the haze image I(x). According to equation (6), the approximate solution of t(x) can be deduced as follows:

[0092]

[0093] The parameter ω is further introduced to control the degree of dehazing, as shown below:

[0094]

[0095] Each image has a 2 / 3 probability of being randomly added with a certain type of fog or converted to a low-light image, and then input into the network for training. Based on normal and synthetic low-quality training data, ShuffleNet V2 learns an appropriate differentiable image processing module in a weakly supervised manner to obtain the processed image, which is then input into YOLOv5 for training to obtain the final target detection model. Therefore, ShuffleNet V2 performs weak supervision through detection loss without manually labeling real images.

[0096] From image enhancement to target detection, the close combination of ShuffleNet V2, DIP, and YOLOv5 solves the problem that images captured in bad weather and low visibility conditions have low visibility due to interference from weather-specific information.

[0097] S3, vehicle target detection: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle. Specifically include:

[0098] S301, load the trained YOLOv5 model from the local, and initialize and configure the YOLOv5 model;

[0099] S302, model training, learning DIP, ShuffleNetV2 and YOLOv5 backbone detection networks in an end-to-end manner; training to obtain a weight file that conforms to the scene in the highway tunnel, and the weight file can realize vehicle target detection in the tunnel;

[0100] S303, model prediction, the original image is input into YOLOv5 for prediction after image preprocessing, and the detection result is returned.

[0101] The YOLOv5 model is a more effective target detection model. After a lot of training and optimization, the YOLOv5 model reduces the amount of calculation and improves the processing speed, which is crucial for highway tunnel environments that require real-time monitoring. YOLOv5 has multi-scale detection capabilities. Vehicles in the tunnel may appear at different sizes due to their distance from the camera. YOLOv5 can detect targets of different sizes and ensure that vehicles of all sizes can be effectively identified. YOLOv5 is easy to deploy. The YOLOv5 model is easy to deploy on different hardware platforms, including edge devices, which is very useful for monitoring systems that may require real-time processing on-site in the tunnel.

[0102] S4. Vehicle Re-ID: The purpose of vehicle re-identification is to identify the same object in different images, which is crucial for multi-camera vehicle tracking. Figure 4 As shown in the figure, in this task, the goal is to further extract features from the target box in S3 to obtain its feature vector. By sending the Re-id vector, the position and confidence value of the bounding box to the single camera tracker, it determines an identifier (ID) for each box. It includes using the trained Resnet101-IBN model to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle; then classify the feature vectors through the trained SVM classifier to obtain a more detailed classification result of vehicle type and color (that is, the feature vector obtained by Re-id). The details are as follows:

[0103] S401, data set creation, annotating the types of vehicles in the images (sedan, suv, van, hatchback, mpv, pickup, bus, truck, estate, sportscar, RV) and colors (yellow, orange, green, gray, red, bule, white, golden, brown, black, purple, pink);

[0104] S402, locally load the ResNet101-IBN model, and initialize and configure the ResNet101-IBN model;

[0105] S403, model training, training the data set in S401 to obtain a weight file that conforms to the scene in the highway tunnel;

[0106] S404, feature vector extraction, the model trained in S403 performs feature extraction on the input video image to obtain a feature vector of each target vehicle in each frame;

[0107] S405, feature vector classification, using SVM as a classifier, and training the SVM using the feature vector in S404;

[0108] S406, identification of vehicle type and color, using the trained SVM classifier to classify the feature vector to obtain a more detailed classification result of vehicle type and color.

[0109] S5. Single-camera vehicle target tracking: Using the high-quality detection results provided by the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is finally obtained, enabling it to handle multi-target tracking tasks in real time.

[0110] In order to solve the ID switching problem after the target disappears temporarily, the present invention proposes an improved target tracking algorithm based on ByteTrack. The traditional ByteTrack tracking algorithm uses the intersection-and-union ratio for trajectory matching when processing low-confidence detection frames. The present invention adds the feature vector after Re-id as the trajectory matching of the low-confidence detection frame on the basis of the ByteTrack tracking algorithm. Compared with the traditional ByteTrack tracking algorithm, the feature vector output by the ResNet101-IBN model can obtain the static features of the target in more detail, and the ID will be more stable during vehicle tracking. The improved ByteTrack is combined with YOLOv5, and YOLOv5 is used for target detection. The improved ByteTrack uses ResNet101-IBN for feature extraction and combines trajectory prediction to achieve target tracking, and optimizes the ID switching problem that occurs when the target disappears temporarily. More accurate and stable single-camera vehicle tracking is achieved. Specifically included are as follows:

[0111] S501, detecting and identifying detection frames with different confidence levels (0-1) based on the YOLOv5 model;

[0112] S502: For each frame of the detection image, the next frame position of the target in that frame is predicted and a prediction frame is generated. At the frame 1 moment, all activated and inactivated trajectories are predicted to generate prediction frames, and the trajectory prediction frame is intersected by the high confidence detection frame at the frame 2 moment by IOU calculation;

[0113] S503, 0.5 is used as the IOU threshold. When it is greater than or equal to 0.5, it is considered a successful match. At this time, the target appears at both frame1 (current frame) and frame2 (next frame) and is identified as the same target, and the trajectory of the target is updated;

[0114] S504: The high-confidence target frame that failed to match is identified as a new track, that is, it appears in the video for the first time. At this time, the new track is identified as an inactive state and needs to be matched three or more times in a row to be transformed into an active state (including the track newly created by the target frame in the first frame);

[0115] S505. The unmatched trajectory will be matched with the low-confidence detection frame of frame2. The matching rule is the feature vector obtained by Re-id. The matching is performed by calculating the cosine similarity of the feature vector, where the threshold of the cosine similarity is set to 0.5. For the matching of the low-score detection frame trajectory, the present invention uses the ResNet101-IBN network to extract features of the target. This feature extraction is based on the detection results of YOLOv5, and a more detailed static feature extraction is performed on the selected target. Finally, the feature extraction vector is used to perform similarity calculation with the unmatched trajectory, so as to match the trajectory of the low-score detection frame (low-confidence detection frame);

[0116] S506, updating the successfully matched target trajectory;

[0117] S507, the low confidence target box that fails to match will be discarded;

[0118] S508, the trajectories that failed to match include the activated state and the inactivated state, and the trajectories in the two states are further judged;

[0119] S509, if the active track has been mismatched 30 times, then delete this track, indicating that the target frame has left the camera monitoring range; by setting the number of consecutive mismatches to retain the track of the target that disappears briefly and combining it with the ResNet101-IBN feature extraction result, the problem of switching when the target that disappears briefly reappears is solved;

[0120] S510: If the secondary matching of an inactive track fails, the track is directly deleted.

[0121] like Figure 5 As shown in the figure, the whole process clearly shows the matching of the objects in the previous and next frames of the video. By re-iding the low-confidence detection boxes, the problem of switching when the objects that disappeared briefly reappeared was solved, and finally high-quality object tracking was achieved.

[0122] S6. Cross-camera vehicle target tracking: Based on hierarchical clustering and constraints, the trajectories under different cameras are matched to achieve the tracking of the same target under different cameras. A multi-camera trajectory matching algorithm based on hierarchical clustering and constraints is proposed. The feature vector after Re-id is used to calculate the similarity between different trajectories, and hierarchical clustering is performed based on the similarity value. In order to improve the accuracy of clustering, constraints between cameras are introduced, including spatial constraints and temporal constraints of the cameras, and are defined by three matrices.

[0123] The details include:

[0124] S601, each target each multi-camera track T i All of them are composed of multiple single camera trajectories τ iThe similarity of the trajectory under each camera is calculated by the cosine similarity between the mean features, where K is the number of features in the aggregation (the number of features extracted by the Re-ID model of the present invention is 2048). The similarity of the trajectory of the target under each camera is calculated using the following formula:

[0125]

[0126] S602. Since the cameras in the highway tunnel are arranged in sequence, the time constraint between cameras can be set as the conversion time between adjacent cameras, instead of providing the conversion time between two cameras, which greatly reduces the calculation amount of the algorithm. Therefore, the time is used as the constraint condition to achieve trajectory matching under different cameras, and the time constraint between cameras is set as the conversion time between adjacent cameras T1 and T2.

[0127] The constraints between cameras can be defined using three matrices. The first is the camera compatibility matrix C, whose elements C ij The value is 1 or 0 depending on whether the two cameras are compatible (or incompatible). That is, whether the video data captured by different cameras can be effectively integrated and associated. The other two matrices are T min and T max , they give a time window in which a transition between camera pairs may occur. If the trajectory of camera τ1 and j th The camera trajectory τ2 is the matching target to be studied, so τ2 should be measured from the departure time of τ1 to the time interval [(T min ) ij , (T max ) ij ] contains j th camera.

[0128] S603 , clustering the trajectories of the same target under different cameras based on the similarity between trajectories and the constraints between cameras.

[0129] Three variants of hierarchical clustering are defined, depending on the definition of T i Multi-camera trajectories. The first is a single-connection approach, where the similarity between the most similar single-camera trajectories is considered as the similarity between two multi-camera trajectories, as shown in Equation 12.

[0130]

[0131] The second is the average connection approach, where the similarity of two multi-camera target tracking trajectories is defined as the average of the similarities between the corresponding single-camera target tracking trajectories, as shown in Equation 13.

[0132]

[0133] The third is a fully connected approach. The similarity between two multi-camera target tracking trajectories is shown in Formula 14, which is the minimum similarity between the corresponding single-camera target tracking trajectories.

[0134]

[0135] The embodiment of the present invention selects an average connection method to cluster the trajectories, so as to achieve tracking of the same target under different cameras.

[0136] The combination of hierarchical clustering between trajectories and constraints between cameras makes the identification of the same target unique under different cameras and ensures the stability of cross-camera target tracking. Even if the appearance captured by the cameras is different (the front, side, and rear of the car), the matching range can be narrowed through constraints, thereby improving the accuracy of cross-camera target tracking.

[0137] When detecting vehicles at intersections and tracking targets across cameras, the steps are the same as those in the highway tunnel mentioned above. The difference is that since the cameras at the intersection are not arranged in sequence, it is necessary to set the time range for each camera to switch between each other. Even if there are highly similar trajectories, they must conform to the time interval for camera switching. The amount of calculation in this scenario is slightly greater than when the cameras are arranged in sequence. The pseudo code for the hierarchical clustering algorithm and the pseudo code for the camera compatibility test function are shown in the figure below. Figure 6 and Figure 7 shown.

[0138] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a cross-camera vehicle target tracking system in a highway tunnel based on deep learning. The system can realize the functions provided by the above method embodiment. The system uses a Linux server and runs an algorithm program on the server to process the images captured by the camera. The generated vehicle detection and tracking results are transmitted to the Web end in the form of JSON data stream via HTTP, which is convenient for data statistics and management. Figure 8 As shown, the system includes the following modules:

[0139] Image acquisition module 110: Install multiple cameras at different positions in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames of the video images through OpenCV, import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color, and form a data set;

[0140] Image preprocessing module 120: construct an image preprocessing model, and preprocess the video image through the image preprocessing model;

[0141] Vehicle target detection module 130: uses the trained YOLOv5 model to detect and identify the preprocessed image, identifies the vehicle in the image and generates a detection box for the vehicle, and then marks the type and color of the vehicle;

[0142] Vehicle re-identification module 140: uses the trained Resnet101-IBN model to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle; then classifies the feature vectors through the trained SVM classifier to obtain more detailed classification results of vehicle type and color;

[0143] Single-camera vehicle target tracking module 150: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained;

[0144] Cross-camera vehicle target tracking module 160: matches the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

[0145] The system of the present invention has significant application value in highway tunnel environments where traffic safety requirements are high and edge devices are required for real-time monitoring. When the environment in the tunnel is relatively complex and there are challenges such as poor visibility or heavy traffic, when the existing vehicle detection and tracking methods are insufficient in accuracy, real-time performance, flexibility and practicality, and when the safety management level of the tunnel has not yet reached the expected standards, the use of this system can significantly improve the efficiency and safety of vehicle target detection and cross-camera tracking in highway tunnels. Through the application of this system, continuous and accurate monitoring of vehicles in the tunnel can be achieved, and potential traffic anomalies can be warned in a timely manner, thereby enhancing the traffic management capabilities of the tunnel.

[0146] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides an electronic device, such as Fig. 9 As shown, the device includes: a processor 210; and a memory 220, for storing one or more programs;

[0147] When the one or more programs are executed by the processor 210, the processor executes the deep learning-based cross-camera vehicle target tracking method in a highway tunnel.

[0148] The method for tracking vehicle targets across cameras in a highway tunnel based on deep learning specifically includes the following steps:

[0149] Image acquisition: Install cameras at different locations in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color to form a data set;

[0150] Image preprocessing: Build an image preprocessing model and preprocess the video image through the image preprocessing model;

[0151] Vehicle target detection: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle;

[0152] Vehicle Re-ID: The trained Resnet101-IBN model is used to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle. The trained SVM classifier is then used to classify the feature vectors to obtain more detailed classification results for vehicle type and color.

[0153] Single-camera vehicle target tracking: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained;

[0154] Cross-camera vehicle target tracking: Match the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

[0155] Based on the same inventive concept as the above-mentioned method embodiment, the embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 210, the method for cross-camera vehicle target tracking in a highway tunnel based on deep learning is implemented.

[0156] The deep learning-based cross-camera vehicle target tracking method in a highway tunnel specifically includes the following:

[0157] Image acquisition: Install cameras at different locations in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color to form a data set;

[0158] Image preprocessing: Build an image preprocessing model and preprocess the video image through the image preprocessing model;

[0159] Vehicle target detection: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle;

[0160] Vehicle Re-ID: The trained Resnet101-IBN model is used to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle. The trained SVM classifier is then used to classify the feature vectors to obtain more detailed classification results for vehicle type and color.

[0161] Single-camera vehicle target tracking: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained;

[0162] Cross-camera vehicle target tracking: Match the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

[0163] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0165] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk. It should be noted that, in this article, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0166] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0167] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0168] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0169] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0170] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for tracking vehicle targets across cameras in a highway tunnel based on deep learning, characterized in that: The steps include: S1. Image acquisition: Install multiple cameras at different positions in the highway tunnel, collect video images in the highway tunnel through the cameras, extract frames from the video images through OpenCV, and import each frame of the image into the labelimg annotation platform to annotate the vehicle type and color to form a data set; S2, image preprocessing: construct an image preprocessing model, and preprocess the video image through the image preprocessing model; S3, vehicle target detection: Use the trained YOLOv5 model to detect and identify the preprocessed image, identify the vehicle in the image and generate a detection box for the vehicle, and then mark the type and color of the vehicle; S4, Vehicle Re-ID: Use the trained Resnet101-IBN model to extract features from the detection box output by YOLOv5 to obtain the feature vector of each target vehicle; then use the trained SVM classifier to classify the feature vectors to obtain more detailed classification results of vehicle type and color; S5, single camera vehicle target tracking: Based on the detection results of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained; S6. Cross-camera vehicle target tracking: Match the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

2. The method for tracking vehicle targets across cameras in highway tunnels based on deep learning according to claim 1 is characterized in that: The image preprocessing model includes a lightweight CNN-based parameter predictor ShuffleNetV2 and a differentiable image processing module DIP; the lightweight CNN-based parameter predictor ShuffleNetV2 consists of a ShuffleNetV2 downsampling module, three ShuffleNetV2 feature extraction networks and two fully connected layers; the differentiable image processing module DIP consists of six differentiable filters for dehazing, white balance, gamma, contrast, hue and sharpening.

3. The method for tracking vehicle targets across cameras in highway tunnels based on deep learning according to claim 1, characterized in that: In step S2, it specifically includes: downsampling the video image in the data set through the ShuffleNetV2 downsampling module, and then extracting features through the ShuffleNet V2 feature extraction network and inputting them into the differentiable image processing module DIP, performing dehazing, white balance, gamma, contrast, hue and sharpening processing on the image to obtain the preprocessed image.

4. The method for tracking vehicle targets across cameras in highway tunnels based on deep learning according to claim 3 is characterized in that: The downsampling process includes: copying the input image into two copies; performing 3x3, stride=2, and 1x1 convolution on one copy; performing 1x1, 3x3, stride=2, and 1x1 convolution on the other copy; and finally splicing the outputs of the two branches in terms of the number of channels to complete the downsampling; The feature extraction process includes: dividing the downsampled channel into two parts; performing 3x3, stride=2, and 1x1 convolution on one part, and not processing the other part; performing channel shuffling processing on the outputs of the two parts to complete feature extraction.

5. The method for tracking vehicle targets across cameras in highway tunnels based on deep learning according to claim 1, characterized in that: In step S5, the specific steps include: S501, detecting and identifying detection frames with different confidence levels based on the YOLOv5 model; S502, predict all active and inactive trajectories at frame 1 to generate a prediction frame, and perform an IOU calculation on the trajectory prediction frame and the high confidence detection frame at frame 2; S503, 0.5 is used as the IOU threshold. When it is greater than or equal to 0.5, it is considered a successful match. At this time, the target appears at both frame 1 and frame 2 and is identified as the same target. The trajectory of the target is updated. S504: The high-confidence target box that failed to match is identified as a new track, that is, it appears in the video for the first time. At this time, the new track is identified as an inactive state and needs to be matched three or more times in a row to become active. S505, the unmatched trajectory will be matched with the low confidence detection frame of frame2, and the matching rule is the feature vector obtained by Re-id, and the matching is performed by calculating the cosine similarity of the feature vector, where the threshold of the cosine similarity is set to 0.5; S506, updating the successfully matched target trajectory; S507, the low confidence target box that fails to match will be discarded; S508, the trajectories that failed to match include the activated state and the inactivated state, and the trajectories in the two states are further judged; S509: If the active track has been mismatched 30 times, the track is deleted, indicating that the target frame has left the camera monitoring range; S510: If the secondary matching of an inactive track fails, the track is directly deleted.

6. The method for tracking vehicle targets across cameras in highway tunnels based on deep learning according to claim 1, characterized in that: In step S6, the specific steps include: S601, each target each multi-camera track T i All of them are composed of multiple single camera trajectories τ i The similarity of the trajectory under each camera is calculated by the cosine similarity between the mean features. The similarity of the trajectory of the target under each camera is calculated by the following formula: S602, using time as a constraint condition to achieve trajectory matching under different cameras, setting the time constraint between cameras to the conversion time between adjacent cameras τ1 and τ2; S603, clustering the trajectories of the same target under different cameras by similarity between trajectories and constraints between cameras, selecting an average connection method to cluster the trajectories, and realizing tracking of the same target under different cameras, wherein the average connection clustering method is expressed as follows:

7. A system for a vehicle target tracking method across cameras in a highway tunnel based on deep learning according to any one of claims 1 to 6, characterized in that: Includes the following modules: Image acquisition module (110): installing multiple cameras at different positions in the highway tunnel, collecting video images in the highway tunnel through the cameras, extracting frames from the video images through OpenCV, and importing each frame of the image into the labelimg annotation platform to annotate the vehicle type and color, thereby forming a data set; Image preprocessing module (120): constructing an image preprocessing model, and preprocessing the video image through the image preprocessing model; Vehicle target detection module (130): uses the trained YOLOv5 model to detect and identify the preprocessed image, identifies the vehicle in the image and generates a detection box for the vehicle, and then marks the type and color of the vehicle; Vehicle re-identification module (140): uses the trained Resnet101-IBN model to extract features from the detection box output by YOLOv5 to obtain a feature vector for each target vehicle; then uses the trained SVM classifier to classify the feature vectors to obtain a more detailed classification result of vehicle type and color; Single camera vehicle target tracking module (150): based on the detection result of the YOLOv5 model and the feature vector obtained by Re-id combined with the improved ByteTrack tracking algorithm, the trajectory of the target under the camera is obtained; Cross-camera vehicle target tracking module (160): matches the trajectories under different cameras based on hierarchical clustering and constraints to achieve tracking of the same target under different cameras.

8. An electronic device, characterized in that: The electronic device comprises: a processor (210); and a memory (220) for storing one or more programs; When the one or more programs are executed by the processor (210), the processor executes the deep learning-based cross-camera vehicle target tracking method in a highway tunnel as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by the processor (210), the method for tracking vehicle targets across cameras in a highway tunnel based on deep learning is implemented as described in any one of claims 1 to 6.

Images