A vehicle target detection method applicable to highway surveillance cameras

Abstract

This invention discloses a vehicle target detection method suitable for highway surveillance cameras, comprising the following steps: acquiring a vehicle target detection dataset from highway surveillance video; constructing a backbone network; constructing a target detection subnetwork; designing a data augmentation strategy; and training the highway surveillance video vehicle target detection network based on the SSD algorithm. This invention's vehicle target detection method for highway surveillance cameras exhibits strong feature extraction capabilities, high accuracy in detecting small target vehicles, and good localization performance, and can be directly applied to vehicle target detection in highway scenarios.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a vehicle target detection method suitable for highway surveillance cameras. Background Technology

[0002] Vehicle target detection in highway scenarios is a crucial component of intelligent highway systems, tasked with identifying and locating vehicle targets in images or videos. However, the diversity of terrain, vehicle size and type, as well as various weather conditions and occlusion issues, makes vehicle target detection exceptionally challenging, especially the detection of small vehicles. Small vehicles are particularly difficult to detect due to their limited available features, high positioning accuracy requirements, imbalanced sample sizes, and clustering. Therefore, researching methods for rapidly and accurately identifying vehicle targets in a scene is a hot topic and a major challenge in intelligent highway system research. This research can provide strong technical support for subsequent tasks such as vehicle tracking, congestion detection, parking event detection, and traffic flow prediction, ensuring the reliability of the intelligent highway system.

[0003] Vehicle object detection, as a subtask of object detection, has always been an important research direction in the field of computer vision. With the rapid development of Convolutional Neural Networks (CNNs), a large number of excellent object detection algorithms have emerged in recent years, such as the YOLO series, SSD series, and R-CNN series, which have greatly inspired research on vehicle object detection methods. However, due to the complex traffic environment of highways, directly applying existing object detection algorithms to vehicle object detection in highway scenarios often fails to achieve satisfactory results. Therefore, researching how to improve the accuracy of vehicle object detection in highway scenarios is of great significance for traffic management and travel safety.

[0004] A review of relevant patents reveals that a Chinese patent discloses a vehicle target recognition method based on improved YOLO multi-scale detection (publication number CN116246059A). This method enhances the features of concentrated vehicle regions in the image by adding an ECA attention mechanism to weight the feature map with minimal computational cost. Furthermore, it improves the detection performance of small-scale vehicle targets by adding a small target detection layer to the YOLOv5s network structure. Another Chinese patent discloses a vehicle recognition method based on the multi-scale YOLO algorithm (publication number CN115171074A), which introduces RFBNet to perform multiple fusions of the network's original receptive field and introduces dilated convolutions to increase the receptive field and fuse features of different sizes, thereby improving vehicle recognition accuracy. However, existing technologies do not fully consider the interference of large target information in shallow feature maps on small target detection, nor whether the model has been sufficiently trained for small targets. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a vehicle target detection method suitable for highway surveillance cameras. This invention aims to solve the problems of poor detection performance and inadequate positioning performance of existing vehicle target detection methods for highway surveillance cameras.

[0006] To achieve the above objectives, the present invention provides a vehicle target detection method suitable for highway surveillance cameras, comprising the following steps:

[0007] S1. Obtain the vehicle target detection dataset from highway surveillance video;

[0008] S2. Establishment of the backbone network;

[0009] S3. Construction of the target detection subnetwork;

[0010] S4. Design of data augmentation strategies;

[0011] S5. Training of a vehicle target detection network for highway surveillance video based on the SSD algorithm.

[0012] Furthermore, step S1 includes the following sub-steps:

[0013] S1.1 Collects video from highway surveillance cameras, converts the video into images frame by frame, and uses the LabelImg tool to create a dataset from the images;

[0014] S1.2 Clean the dataset, filter out and remove images with incorrect annotations;

[0015] S1.3 uses the K-means clustering algorithm to design the prior box size;

[0016] S1.4 Randomly select 90% of the images in the dataset as the training set and the remaining 10% as the test set. The training set and the test set are used for training and testing the model, respectively.

[0017] Furthermore, in step S1.1, the dataset includes vehicle training images and annotation information files.

[0018] Furthermore, step S2 includes the following sub-steps:

[0019] S2.1 introduces the RepVGG structural reparameterization network as the feature extraction network for the SSD algorithm;

[0020] The RepVGG network includes stage0, stage1, stage2, stage3, stage4 and an adaptive average pooling layer. Stage0 has 1 convolutional layer, stage1 has 4 convolutional layers, stage2 has 6 convolutional layers, stage3 has 16 convolutional layers, and stage4 has 1 convolutional layer.

[0021] S2.2 Extract feature maps ψ1, ψ2, ψ3 and ψ4 corresponding to stages 1-4, 2-6, 3-16 and 4-1 of the RepVGG network. These feature maps are used as input for subsequent tasks.

[0022] Furthermore, in step S2.1, the RepVGG network adopts a multi-branch structure during the training phase and converts the model into a single-branch structure through a fusion strategy during the inference phase.

[0023] Furthermore, step S3 includes the following sub-steps:

[0024] S3.1 performs three upsampling operations on feature map ψ4 to obtain a new feature map.

[0025] S3.2 Connect feature map ψ3 with Feature fusion is performed to obtain φ3, ψ4 and φ3 for large target vehicle detection in the SSD algorithm;

[0026] S3.3 on feature map ψ1 and Perform feature erasure operation to erase large target information in feature map ψ1, obtaining φ1; and perform feature map ψ2 and... A feature erasure operation is performed to erase the large target information in feature map ψ2, resulting in φ2; φ1 and φ2 are used for small target vehicle detection in the SSD algorithm.

[0027] Furthermore, the specific steps of step S4 are as follows:

[0028] Based on the SSD algorithm, a Copy-Reduce-Paste data augmentation method is introduced to construct a multi-branch data augmentation strategy, thereby increasing the proportion of small targets in the dataset and enabling small target vehicles to be fully trained.

[0029] Furthermore, step S5 includes the following sub-steps:

[0030] S5.1 Design the overall loss function of the SSD algorithm, including the localization loss and confidence loss for target detection;

[0031] The input image size for the S5.2 network is set to 320*320, the initial learning rate is set to 0.0006, and the number of iterations is set to 200 epochs.

[0032] S5.3 sets the optimizer to the Adam optimizer;

[0033] After training with S5.4, the RepVGG network model is converted into a single-branch structure.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention, starting from the actual environment of highways, addresses the problems of poor detection and localization performance of small target vehicles by highway surveillance cameras by designing an improved SSD vehicle target detection method. The SSD algorithm, as a multi-level target detector, extracts feature maps of different sizes through a backbone network to construct a feature pyramid. Shallow feature maps are used to detect small targets, while deep feature maps are used to detect large targets, effectively alleviating the problem of multi-scale target detection. To address the poor feature extraction capability of the backbone network, this invention introduces a structured reparameterization network, PepVGG, to replace the VGG network, enhancing its feature extraction ability. To address the issue that large target information in shallow feature maps interferes with the detection of small target vehicles, a feature erasing network is designed to erase large target information in shallow feature maps, thereby improving the detection accuracy of small target vehicles. To address the problem of the low proportion of small targets in the dataset, a Copy-Reduce-Paste data augmentation method is introduced, and a multi-branch data augmentation strategy is designed to increase the number of small target vehicles. To address the poor localization performance of the SSD algorithm, a Focal-EIOU loss function is introduced to replace the IOU loss function.

[0036] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a highway vehicle target detection method based on the SSD algorithm.

[0038] Figure 2 This is a schematic diagram of the RepVGG network structure;

[0039] Figure 3 This is a diagram illustrating a data augmentation strategy.

[0040] Figure 4 This is a diagram of the target detection subnetwork structure. Detailed Implementation

[0041] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0042] like Figure 1 As shown, the present invention provides a vehicle target detection method suitable for highway surveillance cameras, comprising the following steps:

[0043] S1. Obtain the vehicle target detection dataset from highway surveillance video;

[0044] S2. Establishment of the backbone network;

[0045] S3. Construction of the target detection subnetwork;

[0046] S4. Design of data augmentation strategies;

[0047] S5. Training of a vehicle target detection network for highway surveillance video based on the SSD algorithm.

[0048] Step S1: Obtain the vehicle target detection dataset from highway surveillance video, which includes the following sub-steps:

[0049] S1.1 Collect videos from highway surveillance cameras, convert the videos into frame-by-frame images, and use the LabelImg tool to create a dataset from the images; then convert the dataset into VOC dataset format, which contains three folders: Annotations, ImageSets, and JPEGImages, used to store the image annotation files, image numbers for training and testing, and the original images, respectively;

[0050] S1.2 Cleans the dataset, filters out and removes images with incorrect labels, and further ensures the reliability of training;

[0051] S1.3 Statistically analyze the ground truth bounding boxes of vehicles in the dataset to obtain the size and aspect ratio of the ground truth bounding boxes of vehicles. Then, referencing the K-means clustering algorithm in YOLOv3, obtain a better prior bounding box size and design a reasonable prior bounding box size and aspect ratio based on this.

[0052] S1.4 Randomly select 90% of the images in the dataset as the training set and the remaining 10% as the test set, which are used for model training and testing, respectively.

[0053] Step S2: Backbone network construction, which includes the following sub-steps:

[0054] S2.1 introduces the RepVGG structural reparameterization network as the feature extraction network for the SSD algorithm;

[0055] During the training phase, it adopts a multi-branch structure, which is equivalent to training multiple models at the same time. During the inference phase, without affecting the detection accuracy, the model is converted into a single-branch structure through a fusion strategy, which not only ensures the feature extraction capability of the model, but also improves the inference speed of the model.

[0056] like Figure 2 As shown, the RepVGG network includes stage0, stage1, stage2, stage3, stage4 and an adaptive average pooling layer. Stage0 has 1 convolutional layer, stage1 has 4 convolutional layers, stage2 has 6 convolutional layers, stage3 has 16 convolutional layers, and stage4 has 1 convolutional layer.

[0057] S2.2 A downsampling operation is performed in each stage, and the corresponding feature maps ψ1, ψ2, ψ3 and ψ4 are extracted from the RepVGG network stages 1-4, stage 2-6, stage 3-16 and stage 4-1, respectively. The feature maps ψ1, ψ2, ψ3 and ψ4 are used as inputs for subsequent tasks; the resolution of the feature layers in the RepVGG network is shown in Table 1.

[0058] Table 1

[0059] Convolutional layer Stage 1-4 Stage 2-6 Stage 3-16 Stage 4-1 resolution 80×80 40×40 20×20 10×10

[0060] Step S3: Building the object detection sub-network, such as... Figure 4 As shown, the specific steps include the following:

[0061] S3.1 Perform upsampling of feature map ψ4 three times to obtain a new feature map. Used as input for subsequent tasks;

[0062] S3.2 Connect feature map ψ3 with Channel splicing and fusion are performed to obtain feature map φ3. Feature maps ψ4 and φ3 are used for large target vehicle detection in the SSD algorithm.

[0063] S3.3 will feature map Normalization is performed using the sigmoid function, and then the feature map ψ1 is multiplied by the normalized feature map to obtain the result. Use afterwards Erasing large target information from feature map ψ1 yields φ1; similarly, for feature maps ψ2 and ψ3, the large target information is removed. A feature erasure operation is performed to erase the large target information in feature map ψ2, resulting in φ2; φ1 and φ2 are used for small target vehicle detection in the SSD algorithm.

[0064] The expression for the sigmod function is as follows:

[0065]

[0066] In the formula, x represents the input (here, the pixel value of the feature map).

[0067] Step S4: Design of data augmentation strategies, such as... Figure 3 As shown, the specific steps are as follows:

[0068] Following a series of optical and geometric transformation data augmentation methods in the SSD algorithm, a Copy-Reduce-Paste data augmentation branch is designed. The Copy-Reduce-Paste data augmentation method is selected with a 50% probability, forming a multi-branch data augmentation strategy. This increases the proportion of small targets in the dataset, allowing small target vehicles to be fully trained, thereby enhancing the detection accuracy of small target vehicles.

[0069] Step S5: Training the vehicle target detection network for highway surveillance videos based on the SSD algorithm, specifically including the following sub-steps:

[0070] S5.1 The overall loss function of the SSD algorithm is designed as L total Including the location loss function L conf And confidence loss function L loc The overall loss function is L total The expression is:

[0071]

[0072] In the formula, x represents whether the prior box and the ground truth box match for a certain category; c represents the confidence score for each category; l represents the predicted box; g represents the ground truth box; and N represents the number of prior boxes that match the ground truth target. Used to adjust the ratio between location loss and confidence loss, typically set to 1; L conf L is the confidence loss function; loc This is the location loss function.

[0073] The confidence loss uses the Softmax loss function over multi-class confidence levels. It is a combination of Softmax and Cross-entropy loss. The confidence loss function L... conf The expression is:

[0074]

[0075] In the formula, i represents the index of the prior bounding box; j represents the index of the ground truth bounding box; P represents the category index, and when P=0, it represents the background; This means matching the i-th prior box with the j-th ground truth box, where the category of the ground truth box is P; Let represent the probability of category P corresponding to the i-th prior box; in the formula, the part i∈Pos represents the loss for positive samples, and i∈Neg represents the loss for negative samples.

[0076] The Focal-EIOU loss function is used instead of the original IoU loss function to enhance the localization performance of the SSD algorithm. The position loss function L... loc The expression is:

[0077] L loc =L IoU +L dis +L asp

[0078] In the formula, L loc L represents the total position loss function; IoU L represents the IoU loss function; dis L represents the loss function representing the center distance between the predicted bounding box and the ground truth bounding box; asp The loss function represents the width and height of the predicted bounding box and the ground truth bounding box.

[0079] L IoU The expression is as follows:

[0080] L IoU =1-IoU

[0081] In the formula, IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box.

[0082] L dis The expression is as follows:

[0083]

[0084] In the formula, ρ represents the Euclidean distance between the predicted bounding box and the ground truth bounding box; b represents the center point of the predicted bounding box; b gt d represents the center point of the ground truth box; d represents the diagonal length of the smallest bounding box that covers both the predicted box and the ground truth box.

[0085] L asp The expression is as follows:

[0086]

[0087] In the formula, w and w gt These represent the widths of the predicted bounding box and the ground truth bounding box, respectively; h and h gt These represent the heights of the predicted bounding box and the ground truth bounding box, respectively; wc h represents the width of the smallest bounding box that covers both the predicted and ground truth boxes. c This represents the height of the smallest bounding box that covers both the predicted and ground truth boxes.

[0088] The expression for the Focal-EIOU loss function is:

[0089] L Focal-EIoU =IoU γ L loc

[0090] In the formula, IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box; γ is a hyperparameter used to control the curvature of the curve; L loc This indicates EIoU loss.

[0091] The input image size for the S5.2 network is set to 320*320, the initial learning rate is set to 0.0006, the batch size is set to 16, the number of iterations is set to 200 epochs, the probability of Copy-Reduce-Paste data augmentation is set to 0.5, and the first 70% of the total iterations are set to use Copy-Reduce-Paste data augmentation.

[0092] To ensure the stability of the trained model, the learning rate is adjusted using a warmup strategy for the first three epochs of training, as shown in the following expression:

[0093]

[0094] In the formula, Warm_lr represents the learning rate adjusted through the warm-up strategy; lr min This represents the minimum learning rate, which is set to 6×10 in this paper. -6 ;lr init The initial learning rate is represented by epoch; the number of iterations using the WarmUP strategy is represented by iter; the number of iterations required for each epoch is represented by iter; and the current iteration number is represented by Iter.

[0095] After the WarmUP strategy for the current 3 epochs is completed, the learning rate is decreased using the Cosine Annealing algorithm, the expression of which is as follows:

[0096]

[0097] In the formula, T sum T represents the total number of iterations, set to 200; cur This indicates the current iteration number.

[0098] S5.3 sets the optimizer to the Adam optimizer. The Adam optimizer can adaptively adjust the learning rate based on the historical gradient and squared gradient estimates of each parameter. Furthermore, Adam is not very sensitive to the choice of hyperparameters, so it does not require careful tuning of hyperparameters like other optimizers.

[0099] S5.4 fuses the convolutional and batch normalization (BN) layers of the trained model. The fusion process is as follows:

[0100] The formula for a convolutional layer is:

[0101] Conv(x)=W(x)+B

[0102] In the formula, x represents the input of each convolutional layer; W represents the weights; and B represents the bias.

[0103] The formula for the BN layer is:

[0104]

[0105] In the formula, x represents the input; μ represents the mean; σ 2 γ represents the variance; β and γ represent the parameters learned during training.

[0106] Substituting the results of the convolutional layer into the formula for the BN layer, we get:

[0107]

[0108] Further simplification of the above equation yields:

[0109]

[0110] make:

[0111]

[0112]

[0113] The final result of the fusion is:

[0114] BN(Conv(x))=W fused (x)+B fused

[0115] In the formula, W fused B represents the weights after fusion. fused This indicates the bias after fusion.

[0116] Finally, the 1×1 convolutional kernel and the Idntity branch are fused with the 3×3 convolutional kernel to form a 3×3 single-branch structure, which speeds up the model's inference.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A vehicle target detection method applicable to highway surveillance cameras, characterized in that, Includes the following steps: S1. Obtain the vehicle target detection dataset from highway surveillance video; S2. Establishment of the backbone network; S2.1 introduces the RepVGG structural reparameterization network as the feature extraction network for the SSD algorithm; The RepVGG network includes stage0, stage1, stage2, stage3, stage4 and an adaptive average pooling layer. Stage0 has 1 convolutional layer, stage1 has 4 convolutional layers, stage2 has 6 convolutional layers, stage3 has 16 convolutional layers, and stage4 has 1 convolutional layer. S2.2 Extract feature maps ψ1, ψ2, ψ3 and ψ4 corresponding to stages 1-4, 2-6, 3-16 and 4-1 of the RepVGG network. These feature maps are used as input for subsequent tasks. S3. Construction of the target detection subnetwork; S3.1 Perform upsampling of feature map ψ4 three times to obtain a new feature map. Used as input for subsequent tasks; S3.2 Connect feature map ψ3 with Channel splicing and fusion are performed to obtain feature map φ3. Feature maps ψ4 and φ3 are used for large target vehicle detection in the SSD algorithm. S3.3 will feature map Normalization is performed using the sigmoid function, and then the feature map ψ1 is multiplied by the normalized feature map to obtain the result. Use afterwards Erasing large target information from feature map ψ1 yields φ1; similarly, for feature maps ψ2 and ψ3, the large target information is removed. A feature erasure operation is performed to erase the large target information in feature map ψ2, resulting in φ2; φ1 and φ2 are used for small target vehicle detection in the SSD algorithm. S4. Design of data augmentation strategies; Based on the SSD algorithm, a Copy-Reduce-Paste data augmentation method is introduced to construct a multi-branch data augmentation strategy, thereby increasing the proportion of small targets in the dataset and enabling small target vehicles to be fully trained. S5. Training of a vehicle target detection network for highway surveillance video based on the SSD algorithm.

2. The vehicle target detection method applicable to highway surveillance cameras according to claim 1, characterized in that, Step S1 includes the following sub-steps: S1.1 Collects video from highway surveillance cameras, converts the video into images frame by frame, and uses the LabelImg tool to create a dataset from the images; S1.2 Clean the dataset, filter out and remove images with incorrect annotations; S1.3 uses the K-means clustering algorithm to design the prior box size; S1.4 Randomly select 90% of the images in the dataset as the training set and the remaining 10% as the test set. The training set and the test set are used for training and testing the model, respectively.

3. The vehicle target detection method applicable to highway surveillance cameras according to claim 2, characterized in that: In step S1.1, the dataset includes vehicle training images and annotation information files.

4. The vehicle target detection method applicable to highway surveillance cameras according to claim 1, characterized in that: In step S2.1, the RepVGG network adopts a multi-branch structure during the training phase and converts the model into a single-branch structure through a fusion strategy during the inference phase.

5. A vehicle target detection method applicable to highway surveillance cameras according to claim 1, characterized in that, Step S5 includes the following sub-steps: S5.1 Design the overall loss function of the SSD algorithm, including the localization loss and confidence loss for target detection; The input image size for the S5.2 network is set to 320*320, the initial learning rate is set to 0.0006, and the number of iterations is set to 200 epochs. S5.3 sets the optimizer to the Adam optimizer; After training with S5.4, the RepVGG network model is converted into a single-branch structure.

Images