An infrared vehicle detection method based on CNN and VIT hybrid structure

Abstract

The invention discloses an infrared vehicle detection method based on a hybrid structure of CNN and VIT, belonging to the field of computer vision technology. In order to solve the problems of low accuracy, slow speed and large network model parameters of the existing vehicle detection method, the method comprises the following steps: step 1, building a network model: comprising a feature extraction module, a Transformer module and a prediction module; step 2, preparing a data set: selecting a FLIR infrared vehicle data set; step 3, training the network model: inputting the data set prepared in step 2 into the network model built in step 1 for training; step 4, selecting a minimized loss function and an optimal evaluation index; step 5, fine-tuning the model: fine-tuning the trained model using the FLIR data set to obtain stable and usable model parameters; finally making the model detection effect better; step 6, saving the model: solidifying the finally determined model parameters, and then when an infrared vehicle detection operation is required, directly inputting an image into the network to obtain a detection result.

Description

Technical Field

[0001] The invention relates to an infrared vehicle detection method based on a CNN and VIT hybrid structure, belonging to the technical field of computer vision. Background Art

[0002] With the development of computer vision technology and the innovation of various hardware technologies, object detection technology based on convolutional neural networks (CNN) has been applied in all aspects of life, especially vehicle detection technology in road scenes, which has become the focus of research. And with the success of Transformer in the field of natural language processing, many researchers have turned their attention to Vision Transformer (VIT). At present, vehicle detection in road scenes is mostly based on visible light. Although this type of detection method has a good accuracy rate under sufficient lighting conditions, it will greatly reduce the detection accuracy rate under insufficient lighting conditions such as night, rainy days, and foggy days. Considering that this technology is related to the life safety of the driver, a vehicle detection method that can be applied to various scenarios is needed. The infrared-based vehicle detection method can meet this requirement. Infrared imaging only relies on the infrared rays emitted by the object itself and does not rely on light, so it is suitable for most scenarios.

[0003] The Chinese patent publication number is "CN114821492A", and the name is "A road vehicle detection system and method based on YOLOv4". The method first inputs the preprocessed image; then, MobileVit is used as the backbone network to extract the features of the input image; the neck includes the spatial pyramid pooling layer SPP and the prediction layer PANet; the positioning loss function of the model is the CDIOU loss function. The detection accuracy obtained by this method is low, the detection speed is slow, the network model parameters are large, and the overall detection effect is poor. Summary of the invention

[0004] In order to solve the problems of low accuracy, slow speed and large network model parameters of existing vehicle detection methods, the present invention provides an infrared vehicle detection method based on a hybrid structure of CNN and VIT. The infrared vehicle detection has better detection effect and has practical application value. At the same time, the method proposed by the present invention has high accuracy, fast detection speed and small network model parameters.

[0005] The solution to the technical problem of the present invention is:

[0006] An infrared vehicle detection method based on a hybrid structure of CNN and VIT includes the following steps:

[0007] Step 1: Build a network model: feature extraction module, used to extract image features; multi-scale fusion module, used to fuse extracted features; Transformer module, used to perform attention calculation on the extracted features; prediction module, used to predict target type and bounding box.

[0008] Step 2, prepare the data set: select the FLIR infrared vehicle data set;

[0009] Step 3, training the network model: input the data set prepared in step 2 into the network model built in step 1 for training;

[0010] Step 4, select the minimized loss function and the optimal evaluation index: By minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained and saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system;

[0011] Step 5, fine-tune the model: Use the FLIR dataset to fine-tune the trained model to obtain stable and usable model parameters, which ultimately makes the model detection effect better;

[0012] Step 6, save the model: solidify the finalized model parameters. When infrared vehicle detection is required, the image can be directly input into the network to obtain the detection result.

[0013] In the step 1, the feature extraction module consists of convolution block 1, convolution block 2, convolution block 3, A pooling, bottleneck module 1, bottleneck module 2, bottleneck module 3, bottleneck module 4, bottleneck module 5, bottleneck module 6 and bottleneck module 7, wherein each convolution block consists of a skip connection, a convolution layer, an activation function and a normalization layer; wherein each bottleneck module consists of an inverted residual structure, a convolution layer and an activation function; the three convolution blocks perform shallow feature extraction on the image; A pooling performs downsampling to extract deep semantic information of the image; the seven bottleneck modules perform dimension reduction on the feature map after up-dimensioning and extracting features.

[0014] In the step 1, the multi-scale fusion module adopts a feature pyramid network, which is composed of three upsampling layers, convolution one, convolution two, convolution three, convolution four, convolution five, convolution six, convolution seven, convolution eight, and M pooling; wherein convolution one, convolution two, convolution three, and convolution four change the number of channels of each layer of the feature extraction network to 256; convolution five, convolution six, convolution seven, and convolution eight perform a smoothing operation on the fused feature map.

[0015] The Transformer module in step 1 is composed of an encoder and a decoder, wherein the encoder is composed of six encoding blocks with the same structure, each encoding block is composed of a residual connection, multi-head attention, a feedforward network and a normalization layer; the decoder is composed of six decoding blocks with the same structure, each decoding block is composed of a residual connection, multi-head attention, a feedforward network normalization layer and a mask layer.

[0016] In step 4, the loss function used in the training process is a combination of cross entropy and position loss; the detection result obtained should be as accurate as possible. The choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model.

[0017] In the step 4, selecting the accuracy as a suitable evaluation indicator during the training process can effectively evaluate the accuracy of the algorithm detection results and measure the role of the infrared vehicle detection network.

[0018] The beneficial effects of the present invention are as follows:

[0019] 1. The present invention proposes a lightweight feature extraction network, which reduces the complexity of the feature extraction network while ensuring the feature extraction capability, thereby reducing the model size and improving the detection speed without affecting the detection accuracy.

[0020] 2. The present invention proposes a feature pyramid network to fuse shallow spatial information and high-level semantic information, so that the fused feature map has richer information, thereby improving the detection accuracy.

[0021] 3. The present invention constructs an infrared vehicle detection method based on a mixture of CNN and Transformer, which fully integrates the generalization ability of CNN and the large model capacity of Transformer, ensures that the model is not redundant while improving the accuracy, can effectively detect vehicle targets from infrared images, and the implementation method is simple and the effect is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of an infrared vehicle detection method based on a hybrid structure of CNN and VIT in the present invention.

[0023] Figure 2 This is a network structure diagram of an infrared vehicle detection method based on a hybrid structure of CNN and VIT in the present invention.

[0024] Figure 3 This is the specific composition of the Transformer module described in the present invention.

[0025] Figure 4 This is a structural diagram of the feature pyramid network described in the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, an infrared vehicle detection method based on a hybrid structure of CNN and VIT, the method specifically includes the following steps:

[0028] Step 1: Build a network model. The entire network consists of a feature extraction module, a multi-scale fusion module, a Transformer module, and a prediction module.

[0029] The feature extraction module consists of convolution block one, convolution block two, convolution block three, A pooling, bottleneck module one, bottleneck module two, bottleneck module three, bottleneck module four, bottleneck module five, bottleneck module six and bottleneck module seven, wherein each convolution block consists of a skip connection, a convolution layer, an activation function and a normalization layer; wherein each bottleneck module consists of an inverted residual structure, a convolution layer and an activation function; the three convolution blocks perform shallow feature extraction on the image; A pooling performs downsampling to extract deep semantic information of the image; the seven bottleneck modules perform dimensionality increase on the feature map to extract features and then perform dimensionality reduction.

[0030] The multi-scale fusion module adopts the feature pyramid network, which consists of three upsampling layers, convolution one, convolution two, convolution three, convolution four, convolution five, convolution six, convolution seven, convolution eight, and M pooling; among them, convolution one, convolution two, convolution three, and convolution four change the number of channels in each layer of the feature extraction network to 256; convolution five, convolution six, convolution seven, and convolution eight perform smoothing operations on the fused feature maps.

[0031] The Transformer module consists of an encoder and a decoder, where the encoder consists of six encoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network and a normalization layer; the decoder consists of six decoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network normalization layer and a mask layer.

[0032] Step 2: Prepare the dataset. The infrared vehicle image dataset used is the FLIR dataset. The images are preprocessed, the size of each image in the dataset is adjusted, and the size of the input image is fixed.

[0033] Step 3: Train the network model. To train the infrared vehicle detection model, input the data set prepared in step 2 into the network model built in step 1 for training.

[0034] Step 4, select the minimized loss function and the optimal evaluation index: By minimizing the loss function of the network output image and label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained and saved. At the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system; during the training process, the appropriate evaluation index selection accuracy and inference speed can effectively evaluate the accuracy of the algorithm detection results and measure the role of the infrared vehicle detection network.

[0035] Step 5, fine-tune the model: fine-tune the trained model to obtain stable and usable model parameters. This will ultimately improve the model detection effect; the FLIR dataset is used in the process of fine-tuning the model parameters.

[0036] Step 6: Save the model and solidify the finalized model parameters.

[0037] Example:

[0038] like Figure 1 As shown, an infrared vehicle detection method based on a hybrid structure of CNN and VIT, the method specifically includes the following steps:

[0039] Step 1: Build a network model.

[0040] like Figure 2 As shown in the figure, the feature extraction module consists of three convolution blocks, A pooling, and seven bottleneck modules, where each convolution block 1, convolution block 2, and convolution block 3 consists of a skip connection, a convolution layer, an activation function, and a normalization layer, and the convolution kernel size is 1×1 and the step size is 1; A pooling implements downsampling operation with a step size of 2. Experiments show that the feature map details obtained by downsampling with a convolution step size of 2 are worse than the feature map details obtained by downsampling with A pooling; bottleneck module 1, bottleneck module 2, bottleneck module 3, bottleneck module 4, bottleneck module 5, bottleneck module 6, and bottleneck module 7 are composed of three convolution layers and activation functions, and the convolution kernel size is 1×1 and 3×3, and the step size is 1. The activation function is defined as follows:

[0041] ReLU6=min(6,max(0,x))

[0042]

[0043] The multi-scale fusion module includes upsampling layer 1, upsampling layer 2, and upsampling layer 3. The upsampling layer uses the nearest neighbor interpolation method. The principle of the nearest neighbor interpolation method is to find the 4 pixel points in the original image based on the floating point coordinates of the pixel points of the target image, and take the original pixel value closest to the pixel point as the value of the point. The definition of the nearest neighbor interpolation method is as follows:

[0044]

[0045] Transformer module such as Figure 3 As shown in the figure, the Transformer module consists of an encoder and a decoder, where the encoder consists of six encoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network, and a normalization layer; the decoder consists of six decoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network normalization layer, and a mask layer. The encoder output is defined as follows:

[0046] Output=LayerNorm(x+SubLayer(x))

[0047] Step 2, prepare the dataset. The infrared vehicle dataset uses FLIR. The FLIR dataset was released in July 2018 to help developers develop and train convolutional neural networks. There are 14,000 images in total, 10,000 of which are from short video clips, and another 4,000 BONUS images are from a 140-second video.

[0048] Step 3, training the network model. Use the annotated images in the data set as labels. The random size and position can be achieved through software algorithms. The purpose of using the annotated images in the data set as labels is to allow the network to learn better feature extraction capabilities and ultimately achieve better detection results.

[0049] Step 4: Select the minimized loss function value and the optimal evaluation index. The network output and label calculate the loss function, and achieve a better fusion effect by minimizing the loss function. The loss function selects cross entropy and position loss. The cross entropy calculation formula is as follows:

[0050]

[0051] The position loss is defined as follows:

[0052]

[0053] The overall loss function is defined as:

[0054] T loss =L+L1

[0055] The number of training times is set to 100, and the number of images input to the network each time is about 8-16. The upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the larger the number of images input to the network each time, the better, making the network more stable. The learning rate of the training process is set to 0.0001, which can ensure fast fitting of the network without causing overfitting of the network. The network parameter optimizer selects the adaptive moment estimation algorithm. The main advantage of this algorithm is that after bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable. The threshold of the loss function value is set to about 0.0003. If it is less than 0.0003, it can be considered that the training of the entire network has been basically completed.

[0056] Step 5: Fine-tune the model. Fine-tune the parameters of the entire network model using the dataset, where the dataset is the FLIR infrared vehicle dataset.

[0057] Step 6, save the model. After the network training is completed, all parameters in the network need to be saved, and then the image to be detected is input into the network to get the detection result. The network has no requirements on the input image size, and any size is acceptable.

[0058] Among them, the implementation of convolution, activation function, and normalization are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0059] The present invention constructs an infrared vehicle detection method based on a hybrid of CNN and Transformer, which can effectively detect vehicle targets from infrared images. The implementation method is simple and has good results, taking into account both detection accuracy and processing speed. By calculating the relevant indicators of the image obtained by the existing method, the feasibility and superiority of the method are further verified. The comparison of the relevant indicators of the prior art and the method proposed in the present invention is shown in Table 1:

[0060] Table 1 Comparison of the effects of the present invention and the existing solutions

[0061]

[0062] It can be seen from the table that the accuracy of the method proposed in the present invention can reach 84.1%, which is an increase of 11.79%, achieving a very good detection accuracy, and the intersection-over-union ratio and recall rate are also greatly improved. These indicators also further illustrate that the method proposed in the present invention has a good infrared vehicle detection effect.

Claims

1. An infrared vehicle detection method based on a hybrid structure of CNN and VIT, characterized in that: The method comprises the following steps: Step 1, build a network model: the entire network includes a feature extraction module for extracting image features; a multi-scale fusion module for fusing extracted features; The Transformer module is used to perform attention calculations on the extracted features; the prediction module is used to predict the target type and bounding box; Step 2, prepare the data set: select the FLIR infrared vehicle data set; Step 3, training the network model: input the data set prepared in step 2 into the network model built in step 1 for training; Step 4, select the minimized loss function and the optimal evaluation index: By minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained and saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system; Step 5: Fine-tune the model: Use the FLIR dataset to fine-tune the trained model to obtain stable and usable model parameters, which ultimately makes the model detection effect better. Step 6, save the model: solidify the finalized model parameters, and when infrared vehicle detection is required, directly input the image into the network to obtain the detection result; In the step 1, the feature extraction module is composed of convolution block 1, convolution block 2, convolution block 3, A pooling, bottleneck module 1, bottleneck module 2, bottleneck module 3, bottleneck module 4, bottleneck module 5, bottleneck module 6 and bottleneck module 7, wherein each convolution block is composed of a skip connection, a convolution layer, an activation function and a normalization layer; wherein each bottleneck module is composed of an inverted residual structure, a convolution layer and an activation function; the three convolution blocks perform shallow feature extraction on the image; A pooling performs downsampling to extract deep semantic information of the image; the seven bottleneck modules perform dimension reduction after upscaling the feature map to extract features; The multi-scale fusion module uses the feature pyramid network, which consists of three upsampling layers, convolution 1, convolution 2, convolution 3, convolution 4, convolution 5, convolution 6, convolution 7, convolution 8, and M-pooling; among them, convolution 1, convolution 2, convolution 3, and convolution 4 change the number of channels in each layer of the feature extraction network to 256; convolution 5, convolution 6, convolution 7, and convolution 8 perform smoothing operations on the fused feature maps; The Transformer module consists of an encoder and a decoder, where the encoder consists of six encoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network, and a normalization layer; The decoder consists of six decoding blocks with the same structure, each of which consists of a residual connection, multi-head attention, a feedforward network normalization layer, and a mask layer.

2. The infrared vehicle detection method based on the hybrid structure of CNN and VIT according to claim 1 is characterized in that: In step 4, the loss function in the training process is selected to use a combination of cross entropy and position loss; the obtained detection result should be as accurate as possible; The choice of loss function affects the quality of the model. It can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model.

3. The infrared vehicle detection method based on the hybrid structure of CNN and VIT according to claim 1 is characterized in that: In the step 4, selecting the accuracy as a suitable evaluation indicator during the training process can effectively evaluate the accuracy of the algorithm detection results and measure the role of the infrared vehicle detection network.

Images