An infrared pedestrian detection method based on attention mechanism
By introducing attention mechanism and depth separation convolution in infrared pedestrian detection, the problem of poor infrared-visible light fusion feature extraction ability is solved, the detection accuracy and generalization ability are improved, false detection and missed detection are reduced, and more efficient is achieved pedestrian testing.
Patent Information
- Application Number
- CN202211394887.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-11-08
AI Technical Summary
The existing infrared pedestrian detection algorithms have problems with low utilization rate of infrared-visible light fusion feature and poor fusion feature extraction capabilities, resulting in low detection accuracy and serious missed detection caused by pedestrian occlusion.
An infrared pedestrian detection method based on attention mechanism was designed. By constructing a pedestrian detection network model containing multiple attention modules, combining convolution blocks and depth separable convolution, using infrared-visible light images to enhance feature extraction and fusion capability to reduce missed detection.
It significantly improves the accuracy of infrared pedestrian detection, reduces the false detection and missed detection caused by pedestrian occlusion, and shows better generalization ability in recognition tasks, exceeding the effect of convolutional neural networks.
Smart Images

Figure CN115620207B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an infrared pedestrian detection method based on an attention mechanism, and belongs to the technical field of computer vision. Background Art
[0002] Pedestrian detection algorithms based on the fusion of infrared and visible light have been widely used in fields such as unmanned driving collision avoidance systems and video tracking. In particular, the detection of pedestrian safety in unmanned driving has attracted much attention. The shortcoming of the commonly used methods is that they only focus on the fused features and do not pay attention to the single-modal features. As a result, when extracting single-modal features, there is no explicit supervision information on the single-modal branch, which makes the feature representation ability of single-modal extraction weak. At the same time, the weak representation ability will reduce the recognition ability of difficult samples, resulting in the problem of serious missed detection of the network. That is, the existing pedestrian detection algorithm has the problem of low utilization rate of infrared-visible light fusion features and poor fusion feature extraction ability.
[0003] To address these two problems, we refer to the common methods in the field of infrared-visible light image fusion and infrared pedestrian detection, and design an infrared pedestrian detection method based on the attention mechanism. In this method, the attention mechanism module is introduced as the feature extraction part of pedestrian detection, and the infrared-visible light image fusion method is used to merge multimodal image information.
[0004] The Chinese patent publication number is "CN114694001A", and the name is "A method and device for target detection based on multimodal image fusion". This method uses a pure Transformer structure to form a target detection model. First, two global features are extracted through the feature encoding module, and then the feature fusion module fuses the above features. Finally, the prediction module composed of a fully connected layer outputs the target category and target position. The fusion module used in this method will cause the loss of detailed information of the infrared-visible light image; at the same time, the feature extraction part composed of pure Transformer cannot solve the problem of pedestrian occlusion at night, and there is a problem of insufficient ability to capture local features, resulting in the inability to obtain more detailed information when detecting pedestrians, and the detection accuracy is low. Summary of the invention
[0005] In order to solve the problem of low pedestrian detection accuracy in existing detection methods, the present invention provides an infrared pedestrian detection method based on an attention mechanism, which significantly improves the infrared pedestrian detection accuracy and significantly reduces false detections and missed detections caused by pedestrian occlusion, and can achieve better results and generalization capabilities in recognition tasks than the convolutional neural network (CNN) model.
[0006] The solution to the technical problem of the present invention is:
[0007] An infrared pedestrian detection method based on an attention mechanism comprises the following steps:
[0008] Step 1, build a pedestrian detection network model: the entire network includes three parts: feature extraction, feature fusion and output head. The feature extraction part consists of convolution block 1, attention module 1, attention module 2, attention module 3 and attention module 4, where convolution block 1 downsamples the input image to reduce the size of the feature map, attention module 1, attention module 2, attention module 3 and attention module 4 use attention mechanism to increase the attention to the image in the window, and extract features from the image in turn with the same structure. The feature fusion part is composed of upsampling layer 1, upsampling layer 2, downsampling layer 1, downsampling layer 2 and downsampling layer 3 connected in sequence to fuse image features at different levels. The output head part consists of output head 1, output head 2 and output head 3, where the three output heads have the same structure, and its structure consists of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6. The detection result is obtained by using convolution layer, regularization layer and activation function on the fused image features;
[0009] Step 2: Establish an infrared pedestrian target detection dataset: Use visible light cameras and infrared cameras to obtain visible light images and infrared images in real time, and input them into a pre-trained infrared-visible light image fusion network to obtain infrared-visible light fusion images, and then annotate them;
[0010] Step 3: training the pedestrian detection network model: input the fused image prepared in step 1 into the network model constructed in step 2 for training;
[0011] Step 4, minimizing the loss function value: By minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained and the model parameters are saved;
[0012] Step 5: Fine-tune the model: Use the public infrared pedestrian detection dataset to train and fine-tune the model to obtain stable and usable model parameters, which ultimately makes the model more effective for fusion.
[0013] Step 6, save the best model: solidify the final model parameters, and when infrared pedestrian detection is required, directly input the image to be detected into the network to obtain the final infrared pedestrian detection result.
[0014] In step 1, convolution block 1 is composed of convolution layer 1, regularization layer, convolution layer 2 and activation function connected in sequence; attention module 1, attention module 2, attention module 3 and attention module 4 are composed of convolution block, residual structure 1, residual structure 2, residual structure 3 and residual structure 4, and the residual structure uses depth separable convolution and adaptive moving window attention mechanism; convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6 are composed of convolution layer, regularization layer and activation function connected in sequence. The convolution kernel size of convolution block 1 is 3×3, and the step sizes of the two convolution layers are 2 and 1 respectively; the convolution kernel of the convolution block in attention module 1, attention module 2, attention module 3 and attention module 4 is 3×3, and the convolution kernel size of depth separable convolution is 1×1 and 3×3; the convolution kernel size of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6 is 3×3, and the step size is 1.
[0015] The infrared-visible light image fusion network pre-trained in step 2 is a DenseFuse image fusion network;
[0016] In step 4, the cross entropy loss function is selected as the loss function during the training process; the selection of the loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the model quality.
[0017] The beneficial effects of the present invention are as follows:
[0018] 1. Use the DenseFuse image fusion network to process infrared-visible light images, which not only speeds up the fusion speed, but also improves the fusion quality of the output image, providing better input images for subsequent detection tasks.
[0019] 2. In attention modules 1, 2, 3, and 4, deep separable convolution, adaptive moving window attention mechanism, and multi-layer perceptron are used to extract features from feature maps, and multiple jump connections are used. This can provide more compact feature representation and richer semantic information, reduce the amount of computation of the attention module, and reduce false detections and missed detections caused by pedestrian occlusion.
[0020] 3. Using two convolution operations in convolution block 1 can enhance the ability of pedestrian detection network to extract shallow features of the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 This is a flow chart of an infrared pedestrian detection method based on an attention mechanism of the present invention.
[0022] Figure 2 This is a network structure diagram of an infrared pedestrian detection method based on an attention mechanism of the present invention.
[0023] Figure 3 This is a structural diagram of the attention module described in the present invention.
[0024] Figure 4 This is the specific composition of each convolution block in convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6 described in the present invention.
[0025] Figure 5 The convolution block 1 described in the present invention is the specific composition of each convolution block in the attention module.
[0026] Figure 6 This is a structural diagram of the output head of the present invention. DETAILED DESCRIPTION
[0027] The present invention is further described in detail below with reference to the accompanying drawings.
[0028] like Figure 1 As shown, an infrared pedestrian detection method based on an attention mechanism comprises the following steps:
[0029] Step 1, construct a pedestrian detection network model: the entire network includes three parts: feature extraction, feature fusion and output head. The feature extraction part consists of convolution block 1, attention module 1, attention module 2, attention module 3 and attention module 4. Convolution block 1 downsamples the input image to reduce the size of the feature map. Attention module 1, attention module 2, attention module 3 and attention module 4 use attention mechanism to increase the attention to the image in the window, and extract features from the image in turn with the same structure. The feature fusion part is composed of upsampling layer 1, upsampling layer 2, downsampling layer 1, downsampling layer 2 and downsampling layer 3 connected in sequence to fuse image features at different levels. The output head part consists of output head 1, output head 2 and output head 3. The three output heads have the same structure, consisting of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6. The detection result is obtained by using convolution layer, regularization layer and activation function on the fused image features. Convolutional block 1 is composed of convolutional layer 1, regularization layer, convolutional layer 2 and activation function connected in sequence; attention module 1, attention module 2, attention module 3 and attention module 4 are composed of convolutional block, residual structure 1, residual structure 2, residual structure 3 and residual structure 4. The residual structure uses depth-separable convolution and adaptive moving window attention mechanism; convolutional block 2, convolutional block 3, convolutional block 4, convolutional block 5 and convolutional block 6 are composed of convolutional layer, regularization layer and activation function connected in sequence. The convolution kernel size of convolutional block 1 is 3×3, and the stride of the two convolution layers is 2 and 1 respectively; the convolution kernel of the convolution blocks in attention module 1, attention module 2, attention module 3 and attention module 4 is 3×3, and the convolution kernel size of depth-separable convolution is 1×1 and 3×3; the convolution kernel size of convolutional block 2, convolutional block 3, convolutional block 4, convolutional block 5 and convolutional block 6 is 3×3 and the stride is 1.
[0030] Step 2: Establish an infrared pedestrian target detection dataset: Visible light images and infrared images are acquired in real time by visible light cameras and infrared cameras, and input into a pre-trained infrared-visible light image fusion network (DenseFuse image fusion network) to obtain infrared-visible light fusion images, which are labeled; in the pre-training process of the pedestrian detection network, the public SCUT FIR pedestrian dataset is used. The dataset images are enhanced, randomly diffracted, and cropped to the input image size as the input of the entire network.
[0031] Step 3: Train the pedestrian detection network model. Input the fused image obtained in step 1 into the network model constructed in step 2 for training.
[0032] Step 4, minimize the loss function value. By minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been trained and the model parameters can be saved. During the training process, the loss function chooses to use a combination of the original loss and the splicing information loss. The purpose is to make full use of the supervisory signal of the positive sample. At the same time, if the positive sample has a high intersection-to-union ratio, then the corresponding contribution of the loss function during the training process will be larger, so that the training can focus on those high-quality samples.
[0033] Step 5: Fine-tune the model: Use the public infrared pedestrian detection dataset to train and fine-tune the model to obtain stable and usable model parameters. The KAIST dataset is used in the process of fine-tuning the model parameters.
[0034] Step 6, save the model: solidify the final model parameters, and when infrared pedestrian detection is required, directly input the image to be detected into the network to obtain the final infrared pedestrian detection result.
[0035] Example:
[0036] like Figure 1 As shown, an infrared pedestrian detection method based on an attention mechanism comprises the following steps:
[0037] Step 1: Build a pedestrian detection network model.
[0038] like Figure 3 As shown in the figure, the entire network consists of three parts: feature extraction, up and down sampling operations, and output head.
[0039] The feature extraction part consists of convolution block 1, attention module 1, attention module 2, attention module 3 and attention module 4. Figure 5 As shown in , convolutional block 1 is composed of convolutional layer 1, regularization layer, convolutional layer 2 and activation function in sequence. The convolution kernel size is 3×3, and the strides of the two convolutional layers are 2 and 1 respectively. Figure 3 As shown in , attention module 1, attention module 2, attention module 3 and attention module 4 are composed of convolutional blocks, residual structure 1, residual structure 2, residual structure 3 and residual structure 4 connected in sequence. The convolutional block structure is as follows Figure 5 As shown in the figure, the convolution block size is 3×3, the stride is 2 and 1. The convolution kernel size of the depthwise separable convolution is 3×3 and 1×1, and the stride is 1. The residual structure 1 is composed of a depthwise separable convolution, a regularization layer, and an adaptive moving window attention. The residual structure 2 is composed of a regularization layer and a multi-layer perceptron. The residual structure 3 is composed of a depthwise separable convolution, a regularization layer, and an adaptive moving window attention. The residual structure 4 is composed of a regularization layer and a multi-layer perceptron.
[0040] The feature fusion part is composed of upsampling layer 1, upsampling layer 2, downsampling layer 1, downsampling layer 2 and downsampling layer 3 connected in sequence to fuse image features at different levels.
[0041] The output head part consists of output head 1, output head 2 and output head 3. The three output heads have the same structure, consisting of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6. They first pass through convolution block 2, and then pass through the connection combination of convolution block 3, convolution block 4, convolution block 5 and convolution block 6 to obtain category output, regression output and object output respectively. Figure 4 As shown in FIG. 1 , the structures of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6 are composed of a convolution layer, a regularization layer and an activation function connected in sequence, and the convolution kernel size is 3×3 and the step size is 1.
[0042] Step 2: Establish an infrared pedestrian target detection dataset. Visible light images and infrared images are obtained in real time by visible light cameras and infrared cameras. SCUTFIR is used in the pre-training process of pedestrian detection network, including 211011 frames of images, a total of 477907 detection frames, and 7659 pedestrians.
[0043] Step 3: Train the pedestrian detection network model. Perform image enhancement on the fused image from step 1, perform random diffraction transformation on the same image, and crop it to the size of the input image as the input of the entire network, and annotate the fused image. The random size and position can be achieved through software algorithms.
[0044] Step 4, minimize the loss function value. The network output and label calculation loss function is to achieve better detection results by minimizing the loss function. The loss function selects the combination of the original loss and the splicing information loss. Each loss is further divided into classification loss and regression box loss. For the loss function, the output of the classification layer and the regression layer are respectively {p i} and {t i} indicates that these two items are respectively represented by N cls and N reg Normalize, where β is the weight factor. During the back propagation process, the parameters of multiple classification and regression boxes are shared, aiming to enable different feature maps to learn more semantic information in addition to label information. The total loss function is defined as:
[0045]
[0046]
[0047] The number of training times is set to 220, and the number of images input to the network each time is about 8-16. The upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the larger the number of images input to the network each time, the better, making the network more stable. The learning rate of the training process is set to 0.0001, which can ensure fast fitting of the network without causing overfitting of the network. The advantage of the network parameter optimizer choosing the adaptive moment estimation algorithm is that after bias correction, each iteration learning rate has a certain range, making the parameters relatively stable. The threshold of the loss function value is set to about 0.0003. If it is less than 0.0003, it can be considered that the training of the entire network has been basically completed.
[0048] Step 5: Fine-tune the model. Fine-tune the parameters of the entire network model using the infrared pedestrian detection dataset, where the dataset is the KAIST dataset.
[0049] Step 6, save the model. After the network training is completed, all parameters in the network need to be saved, and then the detection results can be obtained by inputting the aligned infrared and visible light pedestrian detection images into the network. The network has no requirements on the size of the two input images, and any size is acceptable, but the size of the two images must be consistent.
[0050] Among them, the implementation of convolution layer, depthwise separable convolution, activation function, upsampling layer, downsampling layer, regularization layer, multilayer perceptron, and adaptive moving window attention are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.
[0051] The present invention constructs an infrared pedestrian detection method based on the attention mechanism, which can directly detect pedestrian targets in the image through infrared-light pedestrian images. The feasibility and superiority of the method are further verified by calculating the relevant indicators of the image obtained by the existing method. The relevant indicators of the existing technology and the method proposed in the present invention are compared in Table 1:
[0052] Table 1 Comparison of relevant indicators of the prior art and the method proposed in the present invention
[0053]
[0054] It can be seen from the table that the method proposed in the present invention has higher precision AP and recall rate Recall and lower loss rate MR. These indicators also further illustrate that the method proposed in the present invention has better pedestrian detection effect.
Claims
1. An infrared pedestrian detection method based on attention mechanism, characterized in that: The method comprises the following steps: Step 1: Build a pedestrian detection network model: The entire network consists of three parts: feature extraction, feature fusion, and output head; The feature extraction part consists of convolution block 1, attention module 1, attention module 2, attention module 3 and attention module 4. Convolution block 1 downsamples the input image to reduce the size of the feature map. Attention module 1, attention module 2, attention module 3 and attention module 4 use the attention mechanism to increase the attention to the image in the window, and extract features from the images in turn with the same structure. Convolutional block 1 is composed of convolutional layer 1, regularization layer, convolutional layer 2 and activation function connected in sequence; Attention module 1, attention module 2, attention module 3 and attention module 4 are composed of convolution blocks, residual structure 1, residual structure 2, residual structure 3 and residual structure 4. The residual structure uses depthwise separable convolution and adaptive moving window attention mechanism. The structures of convolutional block 2, convolutional block 3, convolutional block 4, convolutional block 5 and convolutional block 6 are composed of convolutional layer, regularization layer and activation function connected in sequence; The feature fusion part is composed of upsampling layer 1, upsampling layer 2, downsampling layer 1, downsampling layer 2 and downsampling layer 3 connected in sequence to fuse image features at different levels; The output head part consists of output head 1, output head 2 and output head 3. The three output heads have the same structure, which consists of convolution block 2, convolution block 3, convolution block 4, convolution block 5 and convolution block 6. The detection result is obtained by using convolution, regularization and activation function on the fused image features; Step 2: Establish an infrared pedestrian target detection dataset: Use visible light cameras and infrared cameras to obtain visible light images and infrared images in real time, and input them into the pre-trained DenseFuse image fusion network to obtain infrared-visible light fusion images, and then annotate them; Step 3: training the pedestrian detection network model: input the fused image prepared in step 1 into the network model constructed in step 2 for training; Step 4, minimizing the loss function value: By minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters are considered to have been pre-trained and the model parameters are saved; Step 5: Fine-tune the model: Use the public infrared pedestrian detection dataset to train and fine-tune the model to obtain stable and usable model parameters, which ultimately makes the model more effective for fusion. Step 6, save the best model: solidify the final model parameters, and when infrared pedestrian detection is required, directly input the image to be detected into the network to obtain the final infrared pedestrian detection result.
2. According to the infrared pedestrian detection method based on the attention mechanism of claim 1, it is characterized in that: In the step 1, the convolution kernel size of the convolution block one is 3×3, and the step sizes of the two convolution layers are 2 and 1 respectively; the convolution kernels of the convolution blocks in the attention module one, attention module two, attention module three and attention module four are 3×3, and the convolution kernel sizes of the depthwise separable convolutions are 1×1 and 3×3; the convolution kernel sizes of the convolution blocks two, three, four, five and six are 3×3, and the step size is 1.
3. The infrared pedestrian detection method based on attention mechanism according to claim 1 is characterized in that: The infrared-visible light image fusion network pre-trained in step 2 is a DenseFuse image fusion network.
4. The infrared pedestrian detection method based on attention mechanism according to claim 1 is characterized in that: In step 4, the loss function selects the cross entropy loss function; the choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the model quality.
Citation Information
Patent Citations
Target detection method and device based on multi-modal image fusion
CN114694001A
Infrared and visible light image fusion method based on self-attention mechanism
CN111709902A
Target detection method for infrared image
CN114067126A
Traffic person and vehicle non-target detection method for small target
CN114332780A
Infrared and visible light fusion pedestrian detection method based on single-mode enhancement
CN114612937A
