An infrared target detection method based on feature fusion and attention mechanism

By building an infrared object detection network with feature fusion and attention mechanism, the problems of low infrared object detection accuracy and slow speed are solved, and more accurate object detection and real-time performance are achieved in complex environments.

CN114694002BActive Publication Date: 2024-11-01THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210241296.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-11
Publication Date
2024-11-01
Estimated Expiration
2042-03-11

AI Technical Summary

Technical Problem

The existing infrared object detection technology has the problems of low detection accuracy and slow speed, especially in complex environments, where small targets are difficult to accurately detect and inaccurate target positioning.

Method used

Using infrared object detection method based on feature fusion and attention mechanism, the feature extraction ability is enhanced and improved by building a combination network of the backbone network CSPDarknet53, feature fusion network RFB, neck network PANet and attention network CAR2 and head network YOLO Head Pay attention to target characteristics and optimize network parameters in combination with the freeze training method.

Benefits of technology

It improves the accuracy and real-timeness of infrared target detection, and can more accurately locate and identify infrared targets in occlusion, improves detection accuracy and ensures real-timeness of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114694002B_ABST
    Figure CN114694002B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer vision, and discloses an infrared target detection method based on feature fusion and attention mechanism, which mainly solves the problem of low precision in infrared image target detection due to the small number of infrared image features in the prior art. The scheme is: 1) building an infrared target detection network with feature fusion and attention mechanism; 2) building an infrared image data set; 3) using the built data set to train the infrared target detection network with feature fusion and attention mechanism to obtain a trained infrared target detection network; 4) inputting the infrared image to be detected into the trained network for detection and annotation, and outputting the detection result of the infrared target. The present invention enhances the network's ability to identify and locate infrared, ensures the speed of detection, and improves the detection accuracy, and is widely used in the fields of industry, security, transportation, etc.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an infrared target detection method based on feature fusion and attention mechanism, which is suitable for fields such as nighttime autonomous driving, regional security, and body temperature detection. Background Art

[0002] Infrared target detection uses computer vision technology to determine whether there are targets such as people and cars in infrared images, and to give precise positioning. This technology has a wide range of applications in civil fields such as industry, security, and transportation. For example, infrared technology can detect whether a device has poor contact or damage; infrared technology can be applied to unmanned vehicles for all-weather target detection. After more than ten years of development, the research on infrared target detection systems has become more practical in terms of detection accuracy and more real-time in terms of detection speed. In traditional infrared target detection methods, there are three stages: region selection, feature extraction, and classifier classification. Since the region selection strategy is not targeted and has window redundancy, it will lead to misdetection in the subsequent classification process; secondly, the manually designed features are not robust to environmental diversity. For manually designed features in complex environments, there is no way to obtain accurate semantic information, and the detection accuracy is not high. In addition, the infrared image itself has the characteristics of low contrast and low signal-to-noise ratio, which makes the infrared target detection task more difficult. It is easy to cause problems such as missed detection of small targets and inaccurate target positioning.

[0003] Existing infrared target detection methods based on deep learning can be divided into two types: anchor-based and anchor-free, depending on whether there is an anchor or not. Among them, anchor-based methods are divided into one-stage and two-stage. Joseph Redmond proposed a one-stage target detection algorithm YOLOv1 in his paper "You Only Look Once: Unified, Real-Time Object Detection" published at CVPR in 2016. This method treats the object detection task as a regression problem, using a neural network to directly predict the coordinates of the bounding box, the confidence of the object contained in the box, and the probability of the object from an entire image. This method detects objects very quickly, but is more prone to errors in object positioning. A two-stage target detection algorithm Faster R-CNN was proposed in He Kaiming's paper "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks" published at IEEE in 2017. This method consists of two modules, one is the deep fully convolutional network RPN, which is used to generate regional solutions; the second module is the Fast R-CNN detector, which uses the regional solutions generated by RPN for detection. However, the disadvantage of this method is that Faster R-CNN is trained in two steps, resulting in a low speed of target detection and the inability to ensure real-time target detection.

[0004] In the paper "CornerNet: Detecting Objects as Paired Keypoints" published in ECCV in 2018, Hei Law et al. proposed an anchor-free target detection algorithm CornerNet. This method solves the target detection problem as a key point detection problem, that is, the prediction box is obtained by detecting the two key points in the upper left corner and the lower right corner of the target box. Therefore, there is no concept of anchor in the CornerNet algorithm; the training of the entire detection network starts from scratch and is not based on a pre-trained classification model, which allows users to freely design feature extraction networks without being restricted by pre-trained models. However, the semantic ambiguity of this method leads to unstable detection results, and more methods need to be designed to redistribute weights. Summary of the invention

[0005] The purpose of the present invention is to provide an infrared target detection method based on feature fusion and attention mechanism to address the problems of low detection accuracy and slow speed in existing infrared target detection technology, so as to improve the accuracy of infrared target detection and ensure the real-time performance of infrared target detection.

[0006] To achieve the above object, the technical solution of the present invention includes the following:

[0007] An infrared target detection method based on feature fusion and attention mechanism includes the following steps:

[0008] (1) Construct an infrared target detection network with feature fusion and attention mechanism. According to the order of network structure connection, it is: backbone network CSPDarknet53, feature fusion network RFB, neck network PANet, attention network CAR2 and head network YOLO Head. Among them, the backbone network CSPDarknet53 is used to perform preliminary feature extraction on the input infrared image, the feature fusion network RFB and the neck network PANet are used to enhance the feature extraction ability of the network, the attention network CAR2 is used to make the network pay more attention to the features of the target, and the head network YOLO Head is used for the final target recognition and positioning.

[0009] (2) Constructing a dataset: Collecting a FLIR infrared image dataset, which includes infrared image files and annotation files, and dividing the number of infrared image files and the number of annotation files into a training set, a validation set, and a test set according to a set ratio;

[0010] (3) Using the training set and the validation set to train the infrared target detection network with feature fusion and attention mechanism, and obtain the trained infrared target detection network with feature fusion and attention mechanism;

[0011] In step (4), the test set constructed in step (2) is input into the trained infrared target detection network with feature fusion and attention mechanism for detection and annotation, and an infrared image annotated with a target box is output to obtain the detection result of the infrared target.

[0012] Furthermore, the structural relationship of the backbone network CSPDarknet53 in step (1) is as follows:

[0013] Input layer → first convolution layer → second convolution layer → first combination module CSP1 → third convolution layer → fourth convolution layer → second combination module CSP2 → fifth convolution layer → sixth convolution layer → third combination module CSP8 → seventh convolution layer → eighth convolution layer → fourth combination module CSP8 → ninth convolution layer → tenth convolution layer → fifth combination module CSP4 → eleventh convolution layer, where:

[0014] The first to eleventh convolutional layers are all CBM convolutional layers, that is, they are composed of Conv convolution + Bn batch normalization layer + Mish activation function layer;

[0015] The first combination module CSP1 is composed of three CBM convolutional layers with 64 convolution kernel channels and a CSP residual module;

[0016] The second combined module CSP2 is composed of three CBM convolutional layers with 128 convolution kernel channels and two CSP residual modules;

[0017] The third combined module CSP8 is composed of three CBM convolutional layers with 256 convolution kernel channels and eight CSP residual modules;

[0018] The fourth combined module CSP8 is composed of three CBM convolutional layers with 512 convolution kernel channels and eight CSP residual modules;

[0019] The fifth combined module CSP4 is composed of three CBM convolutional layers with 1024 convolution kernel channels and four CSP residual modules.

[0020] The five combined modules are each composed of two branches, the first branch is the first CBM convolution layer, its input is the output of the previous convolution layer, its output is the CSP residual module, and each CSP residual module is connected in sequence, and the output of the last CSP residual module is connected to the second CBM convolution layer; the second branch is the third CBM convolution layer, its input is the output of the previous convolution layer; the output of the entire combined module is the output of the second CBM convolution layer and the output of the third CBM convolution layer concat;

[0021] The concat is to concatenate the feature values ​​of the two branches in the channel dimension;

[0022] The CSP residual module is composed of two CBM convolutional layers connected in sequence, and the output of the second CBM convolutional layer is added to the input of the first CBM convolutional layer as the output of the module;

[0023] The Mish activation function is expressed as: Mish = x × tanh (ln (1 + e x )), where x represents the output of the previous layer and tanh is the hyperbolic tangent function.

[0024] Furthermore, the structural relationship of the feature fusion network RFB in step (1) is: the first branch → the second branch → the third branch → the fourth branch → the fifth branch, where:

[0025] The first branch consists of two 1_DCBL convolutional layers with 128 convolution kernel channels;

[0026] The second branch consists of two 1_DCBL convolutional layers with 64 and 128 convolutional kernel channels respectively and one 2_DCBL convolutional layer with 128 convolutional kernel channels;

[0027] The third branch consists of three 1_DCBL convolutional layers with 64, 96 and 128 convolutional kernel channels respectively and one 3_DCBL convolutional layer with 128 convolutional kernel channels;

[0028] The fourth branch consists of a 1_DCBL convolution layer with a convolution kernel channel number of 2048;

[0029] The fifth branch consists of a 1_DCBL convolution layer with a convolution kernel channel number of 2048, and its input is the concatenation result of the output of the first branch, the second branch, and the third branch;

[0030] The inputs of the first branch, the second branch, the third branch and the fourth branch are all the outputs of the last layer of the backbone network CSPDarknet53 after passing through the CBL structure, and the final output of the feature fusion network RFB is the result of adding the outputs of the fourth branch and the fifth branch;

[0031] The result of the addition is a feature map with 2048 channels and a size of 13*13 pixels;

[0032] The CBL convolution layer consists of Conv convolution + Bn batch normalization + Leaky_relu activation function;

[0033] The 1_DCBL convolution layer consists of Conv convolution layer + Bn batch normalization + Leaky_relu activation function; the 2_DCBL convolution layer consists of dilated 2 dilated convolution + Bn batch normalization + Leaky_relu activation function; the 3_DCBL convolution layer consists of dilated 3 dilated convolution + Bn batch normalization + Leaky_relu activation function;

[0034] The concat is to concatenate the feature values ​​of the three branches in the channel dimension. After concatenation, the number of channels is 384 and the feature map size is 13*13 pixels.

[0035] The leaky_relu activation function is expressed as: Where x represents the output of the previous layer.

[0036] Furthermore, the structural relationship of the neck network PANet in step (1) is: a combination module → b combination module → c combination module → d combination module, where:

[0037] a combination module, consisting of the first stacking module and a convolutional layer. The first stacking module is a stack of the feature layer after the output of the feature fusion network RFB structure passes through the CBL structure and upsampling, and the 26×26×256 effective feature layer output by the ninth convolutional layer of the backbone network CSPDarknet53 passes through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0038] b Combination module, consisting of the second stacking module and a convolutional layer. The second stacking module is a stack of the feature layer after the output of the combination module a has been processed through the CBL structure and upsampled, and the 52×52×128 effective feature layer output by the seventh convolutional layer of the backbone network CSPDarknet53 has been processed through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels.

[0039] c combination module, which is composed of the third stacking module and the convolution layer. The third stacking module is the stacking of the output feature layer of the a combination module and the feature layer after the output downsampling of the b combination module. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0040] d Combination module, consisting of the 4th stacking module and the convolution layer. The 4th stacking module is the stack of the feature layer after the output downsampling of the c combination module and the output feature layer of the feature fusion network RFB. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0041] The CBL convolution layer consists of Conv convolution + Bn batch normalization + Leaky_relu activation function.

[0042] Furthermore, the structural relationship of the attention network CAR2 in step (1) is as follows: the first combination module → the second combination module → the third combination module, the input of the first combination module is the output of the b combination module in the neck network PANet, the input of the second combination module is the output of the c combination module in the neck network PANet, and the input of the first combination module is the output of the d combination module in the neck network PANet, wherein:

[0043] The first combination module, the second combination module and the third combination module are all composed of the first CAR2 residual module → CA attention module → second CAR2 residual module → Bn batch normalization layer → Leaky_relu activation function layer;

[0044] The first CAR2 residual module and the second CAR2 residual module are both CAR2 residual modules, and the CAR2 residual module is composed of two CBL convolutional layers connected in sequence, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer;

[0045] The CA attention module consists of two branches, of which the first branch is composed of two CAR2 residual modules connected in sequence, and the second branch is composed of two branches 2-a and 2-b. The 2-a branch passes through X avg Pool, and the 2-b branch passes through Yavg Pool. Then the outputs of the 2-a branch and the 2-b branch are concat merged into one branch, and 1×1 pixel convolution and Bn batch normalization are performed. Then the output is split and divided into two branches 2-c and 2-d. The 2-c branch is further subjected to 1×1 pixel convolution and Sigmoid function, and the 2-d branch is further subjected to 1×1 pixel convolution and Sigmoid function. Finally, the outputs of the 2-c branch and the 2-d branch are multiplied and added with the output of the first branch.

[0046] The X avg Pool is average pooling along the width of the feature map, and the Y avg Pool is average pooling along the height of the feature map;

[0047] The concat is to superimpose the feature maps of the two branches 2-a and 2-b in the direction of the feature map height or width;

[0048] The split is to split the output stacked feature map of concat in the direction of the feature map height or width;

[0049] The Sigmoid function is expressed as: Where x represents the output of the previous layer.

[0050] Furthermore, in step (1), there are three structures of the head network YOLO Head. The inputs of the three YOLO Heads are the outputs of the first combination module, the second combination module and the third combination module in the attention network CAR2 respectively. Each YOLOHead is composed of a 3×3 convolutional layer and a 1×1 convolutional layer. The 3×3 convolutional layer is composed of a Conv convolutional layer + a Bn batch normalization layer + a Leaky_relu activation function layer, and the 1×1 convolutional layer is composed only of a Conv convolutional layer.

[0051] Furthermore, step 3 specifically includes the following steps:

[0052] (3a) Infrared target detection network environment with feature fusion and attention mechanism;

[0053] (3b) Download the pre-trained weight file of the original object detection network;

[0054] (3c) Input the training set and validation set in the FLIR infrared image dataset;

[0055] (3d) Using the transfer learning method, the pre-trained weight file is loaded on the infrared target detection network with feature fusion and attention mechanism constructed in (1) to obtain the loaded target detection network;

[0056] (3e) Using the training set and validation set constructed in step (2), the target detection network loaded with the pre-trained weight file is trained and validated to obtain a trained infrared target detection network with feature fusion and attention mechanism.

[0057] Furthermore, the environment of the infrared target detection network with feature fusion and attention mechanism configured in step (3a) includes cuda 10.0, cudnn4.1.5, Python3.7, Pycharm2020.1.2, Pytorch1.2.0 and Torchvision=3.4.0.

[0058] Furthermore, in step (3f), the transfer learning method is used to load the pre-trained weight file yolo4_weights.pth on the infrared target detection network with feature fusion and attention mechanism. The parameters of the trained network are obtained from the VOC dataset and loaded onto the infrared target detection network with feature fusion and attention mechanism, so that the infrared target detection network has the ability to recognize the underlying common features.

[0059] Furthermore, the specific steps of step (3e) are:

[0060] A frozen training method is adopted: the training set and the validation set are input into the infrared target detection network with feature fusion and attention mechanism, the model is fitted to the data sample through the training set, the hyperparameters of the model are adjusted through the validation set and the model's ability is preliminarily evaluated; during the training process, the weights of the backbone network CSPDarknet53 are first frozen for training, and more resources are put into training the parameters of other network layers except the backbone network. After the set training time, the frozen part is unfrozen, and all are trained together, and the network training parameters are continuously optimized until the validation set loss function of the infrared target detection network with feature fusion and attention mechanism tends to be stable, that is, when the validation set loss function converges, the training is stopped, and finally a trained infrared target detection network with feature fusion and attention mechanism is obtained.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] First, the present invention constructs a feature fusion and attention mechanism infrared target detection network composed of a backbone network CSPDarknet53, an RFB feature fusion network, a neck network PANet, an attention network CAR2 and a head network YOLO Head, wherein the RFB feature fusion network increases the receptive field through hole convolution, and fuses features of different layers through a multi-branch structure to simulate the receptive field of human vision, thereby enhancing the feature extraction ability of the network and further improving the detection accuracy; at the same time, the present invention replaces the 5*5 pixel convolution in the third branch of RFB with two 3*3 pixel convolutions, reduces model parameters, improves detection speed, and ensures the real-time performance of infrared target detection.

[0063] Second, the present invention uses the attention network CAR2 and applies it to the output feature maps of the neck network PANet at three different scales, thereby allocating more accurate anchor boxes to the feature maps of different scales, thereby improving the accuracy of infrared target detection.

[0064] Third, the present invention uses a frozen training method, which enables the target detection network to quickly and accurately learn the characteristics of the data set, accelerate the training speed, and facilitate better convergence of the loss function, thereby ensuring the accuracy of infrared target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart for realizing the present invention;

[0066] Figure 2 It is a schematic diagram of the feature fusion and attention mechanism network framework in the present invention;

[0067] Figure 3 It is a schematic diagram of the network framework of the feature fusion network RFB in the present invention;

[0068] Figure 4 This is a schematic diagram of the attention network CAR2 network framework in the present invention;

[0069] Figure 5 A diagram showing the training results of the constructed data set using the present invention;

[0070] Figure 6 The figure is an experimental result diagram of infrared target detection on infrared images using the present invention;

[0071] Figure 7 The figure is a graph of the AP (detection accuracy) and FPS (detection speed) experimental results of infrared target detection on infrared images using the present invention; DETAILED DESCRIPTION

[0072] The embodiments and effects of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0073] Reference Figure 1 , the implementation steps of this embodiment are as follows.

[0074] Step 1: Construct an infrared target detection network with feature fusion and attention mechanism.

[0075] Reference Figure 2 , the infrared target detection network with feature fusion and attention mechanism, according to the order of network structure connection, is: backbone network CSPDarknet53, feature fusion network RFB, neck network PANet, attention network CAR2 and head network YOLO Head; among them, the backbone network CSPDarknet53 is used to perform preliminary feature extraction on the input infrared image, the feature fusion network RFB and the neck network PANet are used to enhance the feature extraction ability of the network, the attention network CAR2 is used to make the network pay more attention to the characteristics of the target, and the head network YOLO Head is used for the final target recognition and positioning;

[0076] The key steps are as follows:

[0077] 1.1) Build the backbone network CSPDarknet53 of the infrared target detection network with feature fusion and attention mechanism:

[0078] The structural relationship is: input layer → first convolution layer → second convolution layer → first combination module → third convolution layer → fourth convolution layer → second combination module → fifth convolution layer → sixth convolution layer → third combination module → seventh convolution layer → eighth convolution layer → fourth combination module → ninth convolution layer → tenth convolution layer → fifth combination module → eleventh convolution layer;

[0079] Set the parameters of each module as follows:

[0080] The first to eleventh convolutional layers are all CBM convolutional layers, that is, they are composed of Conv convolution + Bn batch normalization layer + Mish activation function layer;

[0081] The number of channels of the first convolutional layer and the second convolutional layer are 32 and 64 respectively, the convolution kernel is 3×3, and the stride is 1 and 2 respectively;

[0082] The number of channels of the third and fourth convolutional layers are 64 and 128 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0083] The number of channels of the fifth and sixth convolutional layers are 128 and 256 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0084] The number of channels of the seventh and eighth convolutional layers are 256 and 512 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0085] The number of channels of the ninth and tenth convolutional layers are 512 and 1024 respectively, the convolution kernels are 1×1 and 3×3 respectively, and the strides are 1 and 2 respectively;

[0086] The number of channels of the eleventh convolutional layer is 512, the convolution kernel is 3×3, and the stride is 1;

[0087] The first combination module is composed of three CBM convolutional layers and a CSP residual module. The CBM convolutional layer consists of a convolutional layer with 64 channels, a convolution kernel size of 1×1, and a stride of 1, a batch normalization layer, and a Mish activation function layer. The CSP residual module consists of two CBM convolutional layers with 32 and 64 channels, 1×1 and 3×3 convolution kernels, and a stride of 1. The output of the second CBM convolutional layer is connected to the input of the first CBM convolutional layer.

[0088] The second combination module is composed of three CBM convolutional layers and two CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with 128 channels, a convolution kernel size of 1×1, and a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with 64 and 128 channels, 1×1 and 3×3 convolution kernels, and a stride of 1, and the output of the second CBM convolutional layer is connected to the input of the first CBM convolutional layer;

[0089] The third combination module is composed of three CBM convolutional layers and two CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with 256 channels, a convolution kernel size of 1×1, and a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with 128 and 256 channels, 1×1 and 3×3 convolution kernels, and a stride of 1, and the output of the second CBM convolutional layer is connected to the input of the first CBM convolutional layer;

[0090] The fourth combination module is composed of three CBM convolutional layers and two CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a channel number of 512, a convolution kernel size of 1×1, and a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with channel numbers of 256 and 512, convolution kernels of 1×1 and 3×3, and strides of 1, and the output of the second CBM convolutional layer is connected to the input of the first CBM convolutional layer;

[0091] The fifth combination module is composed of three CBM convolutional layers and two CSP residual modules; the CBM convolutional layer is composed of a convolutional layer with a channel number of 1024, a convolution kernel size of 1×1, and a stride of 1, a batch normalization layer, and a Mish activation function layer; the CSP residual module is composed of two CBM convolutional layers with channel numbers of 512 and 1024, convolution kernels of 1×1 and 3×3, and strides of 1, and the output of the second CBM convolutional layer is connected to the input of the first CBM convolutional layer;

[0092] The five combined modules are each composed of two branches. The first branch is the first CBM convolutional layer, whose input is the output of the previous convolutional layer, and whose output is the CSP residual module, and each CSP residual module is connected in sequence, and the output of the last CSP residual module is connected to the second CBM convolutional layer; the second branch is the third CBM convolutional layer, whose input is the output of the previous convolutional layer; the output of the entire combined module is the output of the second CBM convolutional layer and the output of the third CBM convolutional layer concat;

[0093] The concat is to concatenate the feature values ​​of the two branches in the channel dimension;

[0094] The CSP residual module is composed of two CBM convolutional layers, and the output of the second CBM convolutional layer is added to the input of the first CBM convolutional layer as the output of the module;

[0095] The Mish activation function is expressed as: Mish = x × tanh (ln (1 + e x ), where x represents the output of the previous layer, e x represents e raised to the power of x, ln(1+e x ) means base e (1+e x ), and tanh is the hyperbolic tangent function. The Mish activation function allows a smaller negative gradient to flow when the value is negative, ensuring the flow of information without the problem of gradient saturation.

[0096] 1.2) Reference Figure 3 , build an improved feature fusion network RFB of the infrared target detection network with feature fusion and attention mechanism, whose structure is: branch 1 → branch 2 → branch 3 → branch 4 → branch 5;

[0097] Set the parameters of each module as follows:

[0098] The first branch consists of two 1_DCBL convolutional layers with 128 convolution kernel channels connected sequentially;

[0099] The second branch consists of two 1_DCBL convolutional layers with 64 and 128 convolutional kernel channels respectively and a 2_DCBL convolutional layer with 128 convolutional kernel channels connected in sequence;

[0100] The third branch consists of three 1_DCBL convolutional layers with 64, 96 and 128 convolutional kernel channels respectively and a 3_DCBL convolutional layer with 128 convolutional kernel channels connected in sequence;

[0101] The fourth branch consists of a 1_DCBL convolutional layer with a convolution kernel channel number of 2048 connected in sequence;

[0102] The fifth branch consists of a 1_DCBL convolution layer with a convolution kernel channel number of 2048, and its input is the concatenation result of the output of the first branch, the second branch, and the third branch;

[0103] The inputs of the first branch, the second branch, the third branch and the fourth branch are all the outputs of the last layer of the backbone network CSPDarknet53 after the CBL structure, and the final output of the feature fusion network RFB is the result of adding the outputs of the fourth branch and the fifth branch; the result after the addition is a feature map with 2048 channels and a size of 13*13 pixels;

[0104] The 1_DCBL convolution layer consists of a Conv convolution layer + Bn batch normalization + Leaky_relu activation function; the 2_DCBL convolution layer consists of a dilated convolution with a dilation of 2 + Bn batch normalization + Leaky_relu activation function; the 3_DCBL convolution layer consists of a dilated convolution with a dilation of 3 + Bn batch normalization + Leaky_relu activation function; the dilated convolution can increase the receptive field of the model, obtain more additional contextual information to assist detection, and help improve the detection effect of small targets.

[0105] The concat is to concatenate the feature values ​​of the three branches in the channel dimension. After concatenation, the number of channels is 384 and the feature map size is 13*13 pixels.

[0106] The leaky_relu activation function is expressed as: Where x represents the output of the previous layer. The Leaky_relu activation function converges faster when using the gradient descent method, and when x < 0, a non-zero slope is assigned so that the gradient of neurons less than 0 will not be 0.

[0107] 1.3) Build the neck network PANet network of the infrared target detection network with feature fusion and attention mechanism, and its structure is: a combination module → b combination module → c combination module → d combination module;

[0108] Set the parameters of each module as follows:

[0109] a combination module, consisting of the first stacking module and a convolutional layer. The first stacking module is a stack of the feature layer after the output of the feature fusion network RFB structure passes through the CBL structure and upsampling, and the 26×26×256 effective feature layer output by the ninth convolutional layer of the backbone network CSPDarknet53 passes through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0110] b Combination module, consisting of the second stacking module and a convolutional layer. The second stacking module is a stack of the feature layer after the output of the combination module a has been processed through the CBL structure and upsampled, and the 52×52×128 effective feature layer output by the seventh convolutional layer of the backbone network CSPDarknet53 has been processed through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels.

[0111] c combination module, which is composed of the third stacking module and the convolution layer. The third stacking module is the stacking of the output feature layer of the a combination module and the feature layer after the output downsampling of the b combination module. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0112] d Combination module, consisting of the 4th stacking module and the convolution layer. The 4th stacking module is the stack of the feature layer after the output downsampling of the c combination module and the output feature layer of the feature fusion network RFB. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels;

[0113] The CBL convolution layer consists of Conv convolution + Bn batch normalization + Leaky_relu activation function.

[0114] 1.4) Reference Figure 4 , build the attention network CAR2 of the infrared target detection network with feature fusion and attention mechanism, and its structure is: the first combination module → the second combination module → the third combination module;

[0115] Set the parameters of each module as follows:

[0116] The first combination module, whose input is the output of the b combination module in the neck network PANet, consists of the first CAR2 residual module → CA attention module → second CAR2 residual module → Bn batch normalization layer → Leaky_relu activation function layer:

[0117] (1) The first CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer. Its input is the neck network PANet, 52×52×256 feature layer;

[0118] (2) CA attention module, which consists of two branches. The first branch consists of two CAR2 residual modules connected in sequence. The second branch consists of two branches. The 2-a branch (the first branch in the second branch) passes through Xavg Pool, and the feature map is converted to 1×52×256. The 2-b branch (the second branch in the second branch) passes through Y avgPool, the feature map is converted to 52×1×256, and then the outputs of the 2-a branch and the 2-b branch are concat merged into one branch, the feature map is converted to 104×1×256, and 1×1 pixel convolution and Bn batch normalization are performed, and then the output is split and divided into two branches, 2-c and 2-d. The 2-c branch is further subjected to 1×1 pixel convolution and Sigmoid function, and the feature map is converted to 1×52×256. The 2-d branch is further subjected to 1×1 pixel convolution and Sigmoid function, and the feature map is converted to 52×1×256. Finally, the outputs of the 2-c branch and the 2-db branch are multiplied by the output of the first branch;

[0119] (3) The second CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer, and its input is the output of the CA attention module;

[0120] The second combination module, whose input is the output of the c combination module in the neck network PANet, consists of the first CAR2 residual module → CA attention module → second CAR2 residual module → Bn batch normalization layer → Leaky_relu activation function layer:

[0121] (1) The first CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer. Its input is the neck network PANet, 26×26×512 feature layer;

[0122] (2) The CA attention module consists of two branches. The first branch consists of two CAR2 residual modules connected in sequence. The second branch consists of two branches. The 2-a branch (the first branch in the second branch) undergoes Xavg Pool and the feature map is converted to 1×26×512. The 2-b branch (the second branch in the second branch) undergoes Y avgPool, the feature map is converted to 26×1×512, and then the outputs of the 2-a branch and the 2-b branch are concat merged into one branch, the feature map is converted to 52×1×512, and 1×1 pixel convolution and Bn batch normalization are performed, and then the output is split and divided into two branches, 2-c and 2-d. The 2-c branch is further subjected to 1×1 pixel convolution and Sigmoid function, and the feature map is converted to 1×26×512. The 2-d branch is further subjected to 1×1 pixel convolution and Sigmoid function, and the feature map is converted to 26×1×512. Finally, the outputs of the 2-c branch and the 2-d branch are multiplied by the output of the first branch;

[0123] (3) The second CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer, and its input is the output of the CA attention module;

[0124] The third combination module, whose input is the output of the d combination module in the neck network PANet, consists of the first CAR2 residual module → CA attention module → second CAR2 residual module → Bn batch normalization layer → Leaky_relu activation function layer:

[0125] (1) The first CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer. Its input is the neck network PANet, 13×13×1024 feature layer;

[0126] (2) The CA attention module consists of two branches. The first branch consists of two CAR2 residual modules connected in sequence. The second branch consists of two branches. The 2-a branch (the first branch in the second branch) passes through Xavg Pool and the feature map is converted to 1×13×1024. The 2-b branch (the second branch in the second branch) passes through YavgPool and the feature map is converted to 13×1×1024. Then, the outputs of the 2-a branch and the 2-b branch are concat-merged into one branch. The feature map is converted to 26×1×1024 and 1×1 pixel convolution and Bn batch normalization are performed. Then, the output is split and divided into two branches, 2-c and 2-d. The 2-c branch is further subjected to 1×1 pixel convolution and Sigmoid. Function, the feature map is converted to 1×13×1024, the 2-d branch performs 1×1 pixel convolution and Sigmoid function, the feature map is converted to 13×1×1024, and finally the outputs of the 2-c branch and the 2-d branch are multiplied with the output of the first branch;

[0127] (3) The second CAR2 residual module consists of two CBL convolutional layers, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer, and its input is the output of the CA attention module;

[0128] The X avg Pool is average pooling along the width of the feature map, and the Y avg Pool is average pooling along the height of the feature map;

[0129] The concat operation is to superimpose the feature maps of the two branches 2-a and 2-b in the direction of the feature map height or width;

[0130] The split is to split the output stacked feature map of concat in the direction of the feature map height or width;

[0131] The Sigmoid function is expressed as: In the formula, x represents the output of the previous layer. represents e raised to the power of -x.

[0132] 1.5) Build the head network YOLO Head of the infrared target detection network with feature fusion and attention mechanism. The head network YOLO Head has three structures. The inputs of the three YOLO Heads are the outputs of the first combination module, the second combination module, and the third combination module in the attention network CAR2. Each YOLO Head is composed of a 3×3 convolution layer and a 1×1 convolution layer.

[0133] The 3×3 convolution layer consists of Conv convolution layer + Bn batch normalization layer + Leaky_relu activation function layer.

[0134] The 1×1 convolutional layer consists only of Conv convolutional layers.

[0135] The 3×3 pixel convolution layer integrates the obtained features, and the 1×1 pixel convolution layer uses the features to obtain the final output result.

[0136] Step 2: Build the dataset.

[0137] 2.1) Download the FLIR infrared image dataset, which contains an infrared image folder and an annotation folder, with a total of 10,228 images and 10,228 annotation files, including three types of targets: cars, pedestrians, and bicycles. Adjust the image resolution to 416×416 pixels and adjust the annotation file format to .xml format;

[0138] 2.2) The number of images in the FLIR infrared image folder and the number of annotated files in the annotation folder are randomly divided according to the ratio of training set: validation set: test set = 8:1:1. After the division, the training set contains 8182 images and 8182 annotated files, the validation set contains 1023 images and 1023 annotated files, and the test set contains 1023 images and 1023 annotated files;

[0139] Step 3: Train the infrared target detection network with feature fusion and attention mechanism:

[0140] 3.1) Configure the infrared target detection network environment with feature fusion and attention mechanism, including different software such as cuda 10.0, cudnn4.1.5, Python3.7, Pycharm2020.1.2, Pytorch1.2.0, Torchvision=3.4.0;

[0141] 3.2) Download the pre-trained weight file of infrared target detection network with feature fusion and attention mechanism

[0142] yolo4_weights.pth to the root directory where the target detection network code is located;

[0143] 3.3) Input the training dataset and verification dataset in the FLIR infrared image dataset into the target detection network;

[0144] 3.4) Using the transfer learning method, the pre-trained weight file yolo4_weights.pth is loaded on the infrared target detection network with feature fusion and attention mechanism. The parameters of the trained network are obtained from the VOC dataset and loaded on the infrared target detection network with feature fusion and attention mechanism, so that the target detection network has the ability to recognize the underlying common features;

[0145] 3.5) Using the loaded feature fusion and attention mechanism of the infrared target detection network, the training data set is trained and verified by the validation data set, and the implementation is as follows:

[0146] like Figure 5 As shown in the figure, the frozen training method is adopted: the training set and the validation set are input into the infrared target detection network with feature fusion and attention mechanism, the model is fitted with the data sample through the training set, and the hyperparameters of the model are adjusted through the validation set and the model's ability is preliminarily evaluated. During the training process, the weights of the backbone network CSPDarknet53 are first frozen for training, and more resources are put into training the parameters of other network layers except the backbone network, so that both time and resource utilization can be greatly improved. After training for 25 generations, the frozen part is unfrozen, and all are trained together, and the network training parameters are continuously optimized until 50 generations are trained, and the validation set loss function of the infrared target detection network with feature fusion and attention mechanism tends to be stable, that is, when the validation set loss function converges, the training is stopped. Finally, the trained infrared target detection network with feature fusion and attention mechanism is obtained.

[0147] Step 4: Input the test data set into the trained infrared target detection network with feature fusion and attention mechanism for detection and annotation, output the image annotated with infrared targets, and obtain the detection result of infrared targets.

[0148] The effects of the present invention are further described below in conjunction with simulation experiments.

[0149] 1. Experimental conditions:

[0150] The hardware test platform of the simulation experiment of the present invention is: the CPU is Intel (R) Core (TM) i5-9400F, the main frequency is 2.90GHz, the memory is 16GB, the GPU is NVDIA GeForce RTX 2060, and the software platform is: Windows 10 Professional Edition 64-bit.

[0151] 2. Experimental content and results analysis:

[0152] The simulation experiment of the present invention adopts the method of the present invention to train the constructed infrared target detection network with feature fusion and attention mechanism on the constructed infrared image data set. The trained infrared target detection network with feature fusion and attention mechanism is used to detect and annotate the input image containing infrared targets to obtain the detection result of infrared targets.

[0153] Figure 6 The figure is the experimental result of using the present invention to detect targets in infrared images. Figure 6 (a) is the original infrared image. Figure 6 (b) in the figure is the infrared target detection result of the YOLOv4 network. Figure 6 (c) in the figure is the result of infrared target detection using the feature fusion and attention mechanism network. It can be seen from the figure that the feature fusion and attention mechanism network solves the problem of missed detection of infrared targets caused by occlusion and improves the detection accuracy of the target.

[0154] Figure 7 The AP (detection accuracy) and FPS (detection speed) results of target detection on infrared images using the present invention are shown in the figure. It can be seen from the figure that the network with feature fusion and attention mechanism has improved the mAP of three types of targets in infrared images by 1.5% compared with the YOLOv4 network. The FPS is 29.2, which is slightly lower than that of the YOLOv4 network, but it can ensure the real-time performance of target detection.

[0155] In summary, the infrared target detection method based on feature fusion and attention mechanism network proposed in the present invention can better locate and identify infrared targets under occlusion, the accuracy of infrared target detection is improved, and the real-time performance of infrared target detection is guaranteed.

[0156] The above is only a specific example of the present invention to facilitate the understanding of the present invention by those skilled in the art, but the present invention is not limited to the scope of the specific example. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected by the present invention.

Claims

1. An infrared target detection method based on feature fusion and attention mechanism, characterized in that: The steps include: (1) Construct an infrared target detection network with feature fusion and attention mechanism. According to the order of network structure connection, it is: backbone network CSPDarknet53, feature fusion network RFB, neck network PANet, attention network CAR2 and head network YOLO Head. Among them, the backbone network CSPDarknet53 is used to perform preliminary feature extraction on the input infrared image, the feature fusion network RFB and the neck network PANet are used to enhance the feature extraction ability of the network, the attention network CAR2 is used to make the network pay more attention to the features of the target, and the head network YOLO Head is used for the final target recognition and positioning. (2) Constructing a dataset: Collecting a FLIR infrared image dataset, which includes infrared image files and annotation files, and dividing the number of infrared image files and the number of annotation files into a training set, a validation set, and a test set according to a set ratio; (3) Using the training set and the validation set to train the infrared target detection network with feature fusion and attention mechanism, and obtain the trained infrared target detection network with feature fusion and attention mechanism; Step (4) inputs the test set constructed in step (2) into the trained infrared target detection network with feature fusion and attention mechanism for detection and annotation, outputs the infrared image annotated with the target frame, and obtains the detection result of the infrared target; Among them, the structural relationship of the backbone network CSPDarknet53 in step (1) is as follows: Input layer → first convolution layer → second convolution layer → first combination module CSP1 → third convolution layer → fourth convolution layer → second combination module CSP2 → fifth convolution layer → sixth convolution layer → third combination module CSP8 → seventh convolution layer → eighth convolution layer → fourth combination module CSP8 → ninth convolution layer → tenth convolution layer → fifth combination module CSP4 → eleventh convolution layer, where: The first to eleventh convolutional layers are all CBM convolutional layers, which are composed of Conv convolution + Bn batch normalization layer + Mish activation function layer; The first combination module CSP1 is composed of three CBM convolutional layers with 64 convolution kernel channels and a CSP residual module; The second combined module CSP2 is composed of three CBM convolutional layers with 128 convolution kernel channels and two CSP residual modules; The third combined module CSP8 is composed of three CBM convolutional layers with 256 convolution kernel channels and eight CSP residual modules; The fourth combined module CSP8 is composed of three CBM convolutional layers with 512 convolution kernel channels and eight CSP residual modules; The fifth combined module CSP4 is composed of three CBM convolutional layers with 1024 convolution kernel channels and four CSP residual modules. The five combined modules are each composed of two branches, the first branch is the first CBM convolution layer, its input is the output of the previous convolution layer, its output is the CSP residual module, and each CSP residual module is connected in sequence, and the output of the last CSP residual module is connected to the second CBM convolution layer; the second branch is the third CBM convolution layer, its input is the output of the previous convolution layer; the output of the entire combined module is the output of the second CBM convolution layer and the output of the third CBM convolution layer concat; The concat is to concatenate the feature values ​​of the two branches in the channel dimension; The CSP residual module is composed of two CBM convolutional layers connected in sequence, and the output of the second CBM convolutional layer is added to the input of the first CBM convolutional layer as the output of the module; The Mish activation function is expressed as: Mish = x × tanh (ln (1 + e x )), where x represents the output of the previous layer and tanh is the hyperbolic tangent function.

2. The infrared target detection method based on feature fusion and attention mechanism according to claim 1 is characterized in that: The structural relationship of the feature fusion network RFB in step (1) is: branch 1 → branch 2 → branch 3 → branch 4 → branch 5, where: The first branch consists of two 1_DCBL convolutional layers with 128 convolution kernel channels connected sequentially; The second branch consists of two 1_DCBL convolutional layers with 64 and 128 convolutional kernel channels respectively and a 2_DCBL convolutional layer with 128 convolutional kernel channels connected in sequence; The third branch consists of three 1_DCBL convolutional layers with 64, 96 and 128 convolutional kernel channels respectively and a 3_DCBL convolutional layer with 128 convolutional kernel channels connected in sequence; The fourth branch consists of a 1_DCBL convolutional layer with a convolution kernel channel number of 2048 connected in sequence; The fifth branch consists of a 1_DCBL convolution layer with a convolution kernel channel number of 2048, and its input is the concatenation result of the output of the first branch, the second branch, and the third branch; The inputs of the first branch, the second branch, the third branch and the fourth branch are all the outputs of the last layer of the backbone network CSPDarknet53 after passing through the CBL structure, and the final output of the feature fusion network RFB is the result of adding the outputs of the fourth branch and the fifth branch; The final output result of the feature fusion network RFB is a feature map with 2048 channels and a size of 13*13 pixels; The CBL structure consists of Conv convolution + Bn batch normalization + Leaky_relu activation function; The 1_DCBL convolution layer consists of Conv convolution layer + Bn batch normalization + Leaky_relu activation function; the 2_DCBL convolution layer consists of dilated 2 dilated convolution + Bn batch normalization + Leaky_relu activation function; the 3_DCBL convolution layer consists of dilated 3 dilated convolution + Bn batch normalization + Leaky_relu activation function; The concat is to concatenate the feature values ​​of the three branches in the channel dimension. After concatenation, the number of channels is 384 and the feature map size is 13*13 pixels. The leaky_relu activation function is expressed as: Where x represents the output of the previous layer.

3. The infrared target detection method based on feature fusion and attention mechanism according to claim 1 is characterized in that: The structural relationship of the neck network PANet in step (1) is: a combination module → b combination module → c combination module → d combination module, where: a combination module, consisting of the first stacking module and a convolutional layer. The first stacking module is a stack of the feature layer after the output of the feature fusion network RFB structure passes through the CBL structure and upsampling, and the 26×26×256 effective feature layer output by the ninth convolutional layer of the backbone network CSPDarknet53 passes through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels; b Combination module, consisting of the second stacking module and a convolutional layer. The second stacking module is a stack of the feature layer after the output of the combination module a passes through the CBL structure and upsampling and the 52×52×128 effective feature layer output by the seventh convolutional layer of the backbone network CSPDarknet53 after passing through the CBL structure. The convolutional layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels; c combination module, which is composed of the third stacking module and the convolution layer. The third stacking module is the stacking of the output feature layer of the a combination module and the feature layer after the output downsampling of the b combination module. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels; d Combination module, consisting of the 4th stacking module and the convolution layer. The 4th stacking module is the stack of the feature layer after the output downsampling of the c combination module and the output feature layer of the feature fusion network RFB. The convolution layer is a CBL alternating convolution of 1×1 pixels and 3×3 pixels; The CBL structure consists of Conv convolution + Bn batch normalization + Leaky_relu activation function.

4. The infrared target detection method based on feature fusion and attention mechanism according to claim 3 is characterized in that: The structural relationship of the attention network CAR2 in step (1) is: the first combination module → the second combination module → the third combination module, the input of the first combination module is the output of the b combination module in the neck network PANet, the input of the second combination module is the output of the c combination module in the neck network PANet, and the input of the third combination module is the output of the d combination module in the neck network PANet, where: The first combination module, the second combination module and the third combination module are all composed of the first CAR2 residual module → CA attention module → second CAR2 residual module → Bn batch normalization layer → Leaky_relu activation function layer; The first CAR2 residual module and the second CAR2 residual module are both CAR2 residual modules, and the CAR2 residual module is composed of two CBL convolutional layers connected in sequence, and the output of the second CBL convolutional layer is connected to the input of the first CBL convolutional layer; The CA attention module consists of two branches, where the first branch consists of two CAR2 residual modules connected in sequence, and the second branch consists of two branches 2-a and 2-b. The 2-a branch passes through X avg Pool, and the 2-b branch passes through YavgPool. Then the outputs of the 2-a branch and the 2-b branch are concat-merged into one branch, and 1×1 pixel convolution and Bn batch normalization are performed. Then the output is split and divided into two branches 2-c and 2-d. The 2-c branch performs 1×1 pixel convolution and Sigmoid function again, and the 2-d branch performs 1×1 pixel convolution and Sigmoid function again. Finally, the outputs of the 2-c branch and the 2-d branch are multiplied with the output of the first branch. The X avg Pool is average pooling along the width of the feature map, and the Y avg Pool is average pooling along the height of the feature map; The concat is to superimpose the feature maps of the two branches 2-a and 2-b in the direction of the feature map height or width; The split is to split the output stacked feature map of concat in the direction of the feature map height or width; The Sigmoid function is expressed as: Where x represents the output of the previous layer.

5. The infrared target detection method based on feature fusion and attention mechanism according to claim 4 is characterized in that: In step (1), there are three YOLO Head structures. The inputs of the three YOLO Heads are the outputs of the first, second and third combination modules in the attention network CAR2. Each YOLO Head is composed of a 3×3 convolutional layer and a 1×1 convolutional layer. The 3×3 convolution layer consists of a Conv convolution layer + a Bn batch normalization layer + a Leaky_relu activation function layer, and the 1×1 convolution layer consists only of a Conv convolution layer.

6. The infrared target detection method based on feature fusion and attention mechanism according to claim 1 is characterized in that: Step (3) specifically includes the following steps: (3a) Infrared target detection network environment with feature fusion and attention mechanism; (3b) Download the pre-trained weight file of the original object detection network; (3c) Input the training set and validation set in the FLIR infrared image dataset; (3d) Using the transfer learning method, the pre-trained weight file is loaded on the infrared target detection network with feature fusion and attention mechanism constructed in (1) to obtain the loaded target detection network; (3e) Using the training set and validation set constructed in step (2), the target detection network loaded with the pre-trained weight file is trained and validated to obtain a trained infrared target detection network with feature fusion and attention mechanism.

7. The infrared target detection method based on feature fusion and attention mechanism according to claim 6 is characterized in that: The environment of the infrared target detection network with feature fusion and attention mechanism configured in step (3a) includes cuda 10.0, cudnn4.1.5, Python3.7, Pycharm2020.1.2, Pytorch1.2.0 and Torchvision=3.4.

0.

8. The infrared target detection method based on feature fusion and attention mechanism according to claim 6 is characterized in that: In step (3d), the transfer learning method is used to load the pre-trained weight file yolo4_weights.pth on the infrared target detection network with feature fusion and attention mechanism. The parameters of the trained network are obtained from the VOC dataset and loaded onto the infrared target detection network with feature fusion and attention mechanism, so that the infrared target detection network has the ability to recognize the underlying common features.

9. The infrared target detection method based on feature fusion and attention mechanism according to claim 6 is characterized in that: The specific steps of step (3e) are: A frozen training method is adopted: the training set and the validation set are input into the infrared target detection network with feature fusion and attention mechanism, the model is fitted to the data sample through the training set, the hyperparameters of the model are adjusted through the validation set and the model's ability is preliminarily evaluated; during the training process, the weights of the backbone network CSPDarknet53 are first frozen for training, and more resources are put into training the parameters of other network layers except the backbone network. After the set training time, the frozen part is unfrozen, and all are trained together, and the network training parameters are continuously optimized until the validation set loss function of the infrared target detection network with feature fusion and attention mechanism tends to be stable, that is, when the validation set loss function converges, the training is stopped, and finally a trained infrared target detection network with feature fusion and attention mechanism is obtained.

Citation Information

Patent Citations

  • Neural network and target detection and depth prediction method based on neural network

    CN111047630A

  • Real-time football target detection method based on neural network

    CN111832513A

  • Multi-target detection method and system adaptive to multi-band image

    CN112070111A

  • Feature fusion and dense connection based infrared plane target detection method

    WO2020102988A1

  • Method for detecting image target in smart home environment

    WO2021244079A1