Target detection method and device suitable for infrared imaging scene and storage medium

Abstract

The invention discloses a target detection method and device suitable for an infrared imaging scene and a storage medium, and the method comprises the steps: collecting an infrared scene picture through an infrared camera, marking the shot picture, inputting the marked picture into a network, and extracting a multi-layer feature map through a newly constructed global feature extraction network; inputting the extracted feature map into a feature fusion network constructed by a feature pyramid to carry out multi-scale feature fusion; afterwards, in order to retain tiny target information in the shallow feature map, a jump connection mode is adopted, and the original feature map and the fused feature map are fused again to obtain final input. And an output mode of three detection heads is adopted, so that the detection of a large target is ensured, and the detection of a small target is also ensured. Under the infrared scene, the feature extraction of the target is ensured, and the detection accuracy is ensured at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, relates to the recognition and detection of pedestrians and vehicles in infrared images, and is a target detection method, device and storage medium suitable for infrared imaging scenes. Background Art

[0002] Infrared images are obtained by detecting infrared radiation emitted by objects, and can be used for target detection in complex lighting environments. Compared with traditional visible light imaging technology, infrared imaging technology has the advantages of strong penetration, high measurement accuracy, less impact by weather, and strong resistance to external interference. It has been widely used in the fields of autonomous driving, security monitoring, military reconnaissance, and auxiliary medical diagnosis. However, most of the current detection algorithms focus on the visible light field and are not suitable for the infrared imaging field. For example, the two-stage target detection algorithm represented by R-CNN and the one-stage detection algorithm represented by YOLO and SSD have good effects in visible light scenes, but in infrared imaging scenes, they cannot meet the requirements of feature extraction, and their detection effect is poor. The problem of low detection accuracy still exists. Due to the large background interference, low image resolution, and insufficient texture details of infrared images, it is difficult to extract target features, and the problem of low detection accuracy remains to be solved. Summary of the invention

[0003] The problem to be solved by the present invention is that in infrared scenes, the accuracy of the infrared target detection network is low due to the large similarity between the detected target and the image background, low image resolution, insufficient target texture details and other problems.

[0004] The technical solution of the present invention is: a target detection method suitable for infrared imaging scenes, comprising the following steps:

[0005] Step 1: Obtain the infrared image information collected by the infrared camera;

[0006] Step 2: Build a vehicle and pedestrian dataset in infrared scenes;

[0007] Step 3: Build an improved target detection network based on the YOLOv5 model. For the backbone network, build a feature extraction network based on C3 and build a feature extraction network by integrating the global attention mechanism.

[0008] Step 4: The feature map obtained by the backbone network is subjected to feature fusion, and the feature fusion network is the feature pyramid FPN;

[0009] Step 5: Fuse the original feature map of the backbone network with the feature map after the feature fusion network through skip connection to obtain fused feature maps of different scales;

[0010] Step 6: After adding the SK attention mechanism to the fused feature maps of different scales, the maps are input into the detection head for detection and the target detection results are output;

[0011] Step 7: Use the data set of Step 2 to train the target detection network constructed in Step 3 to Step 6, and use the trained target detection network to perform target detection on infrared images.

[0012] Furthermore, in Step 3, the feature extraction network built on the basis of C3 is called GC_3 network, specifically:

[0013] The GC_3 network includes TGC module, CBS module, BottleNeck module and Concat module;

[0014] The TGC module consists of two branches, one of which passes through a 3*3 convolution, then a global average pooling, a 1*1 convolution, a ReLU activation layer and a 1*1 convolution; the other branch passes through a 5*5 convolution, then a 1*1 convolution, a Softmax layer, and then the result is added to the original input, and then passes through a 1*1 convolution, a ReLU activation layer and a 1*1 convolution, and then the two branches are added to get the output.

[0015] The CBS module includes 1*1Conv layer, BN layer, and LeakyReLU activation layer in sequence to construct feature maps;

[0016] BottleNeck consists of two parts, 1*1 and 3*3 Conv. The 1*1 Conv is used to halve the number of feature map channels, and the 3*3 convolution doubles the number of channels. The two Convs are used to increase the network's receptive field and reduce the amount of calculation.

[0017] The GC_3 network consists of three branches. The first branch contains one TGC module, one CBS module, and one BottleNeck module for deep feature extraction. The second branch contains the TGC module and the CBS module for channel adjustment and retaining the original spatial information. The third branch only contains the CBS module. The three branches are then connected through the Concat module.

[0018] After the picture is input into the feature extraction network, feature maps of different sizes are extracted. After a 3*3 Conv, a feature map P1 with a size of 160*160 and 64 channels is obtained; after three GC_3 modules and a 3*3 Conv, a feature map P2 with a size of 80*80 and 128 channels is obtained; after nine GC_3 modules and a 3*3 Conv, a feature map P3 with a size of 40*40 and 256 channels is obtained; after nine GC_3 modules and a 3*3 Conv, a feature map P4 with a size of 20*20 and 512 channels is obtained.

[0019] Furthermore, the feature fusion network of Step 4 is specifically as follows:

[0020] The obtained feature map P4 is pooled by SPP to obtain the feature map Q4. After upsampling Q4, the feature map with a size of 40*40 and a number of channels of 256 is obtained, which is fused with the feature map P3 to obtain a new feature map Q3; the feature map with a size of 80*80 and a number of channels of 128 obtained after upsampling Q3 is fused with the feature map P2 to obtain a new feature map Q2; the feature map with a size of 160*160 and a number of channels of 64 obtained after upsampling Q2 is fused with the feature map P1 to obtain a new feature map Q1.

[0021] Furthermore, Step 5 obtains fusion feature maps of different scales through skip connections:

[0022] Step 5.1: Add the CA attention mechanism after the feature map P1, and then fuse it with the feature map Q1 through a skip connection to obtain the feature map C1;

[0023] Step 5.2: Downsample the feature map C1 to obtain the feature map C2;

[0024] Step 5.3: Add the CA attention mechanism after the feature map P3, and then fuse it with the down-sampled result of the feature map C2 through a skip connection to obtain the feature map C3;

[0025] Step 5.3: Add the feature map CA attention mechanism after the feature map P4, and then fuse it with the down-sampled result of the feature map C3 through a jump connection to obtain the feature map C4.

[0026] Furthermore, in Step 6, the feature maps C1, C3, and C4 are respectively input into the detection head for detection after passing through the SK attention mechanism. The detection head changes the two 3*3 convolutional layers in the detection head of YOLOv5 into one layer.

[0027] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the target detection network of the target detection method suitable for infrared imaging scenes, which is used for target detection in infrared images.

[0028] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the target detection network of the target detection method suitable for infrared imaging scenes is implemented.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] First, the present invention constructs a TGC feature enhancement module, and constructs a GC_3 feature extraction network through the module. Compared with the traditional feature extraction network, the method of the present invention adopts the newly constructed TGC module to model the global situation, so that the model can extract the relationship of the global context and ensure that the global context information is taken into account. At the same time, TGC uses a multi-branch structure to process multi-scale information to help the network obtain more satisfactory feature information, help the network capture more important detail information, generate richer feature representations, avoid feature loss, and improve the robustness of the network. By aggregating different features of multiple branches, the processing capacity of the network bottleneck layer is enhanced;

[0031] Second, the present invention adopts multiple jump connections to fuse the original feature map containing richer information with the feature map containing deeper features after feature fusion, so that the fused feature map contains both shallow position information in the original feature map and deep semantic information, thereby improving the network's ability to process feature information from a global level;

[0032] Third, the present invention solves the problem of difficult infrared image feature extraction and poor detection results to a large extent through improved feature extraction network and multi-level feature fusion network, combined with multiple attention mechanisms. Using the Arrow Optoelectronics infrared open source data set for testing, the accuracy of the present invention is 6.7% higher than that of the existing detection network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention.

[0034] Figure 2 It is a schematic diagram of the improved detection network structure of the present invention.

[0035] Figure 3 This is a schematic diagram of the network structure of the feature extraction network GC_3 of the present invention.

[0036] Figure 4 It is a schematic diagram of the TGC module structure in the GC_3 network of the present invention. DETAILED DESCRIPTION

[0037] The present invention proposes a target detection and recognition method suitable for infrared imaging scenes. The present invention solves the problem of difficulty in extracting target features and low detection accuracy in infrared scenes. The present invention is highly practical and suitable for various scenes such as unmanned driving and monitoring, and can be deployed on a large scale.

[0038] like Figure 1 As shown, the specific implementation process of the present invention is as follows.

[0039] Step 1: Obtain the infrared image information collected by the vehicle-mounted infrared camera.

[0040] Step 2: Construct infrared pedestrian and vehicle datasets, annotate pedestrian and vehicle frames and indicate categories, and divide the classified images into training and test sets in a ratio of 9:1.

[0041] Step 3: Build an improved target detection network based on the YOLOv5 model. For the backbone network, build a feature extraction network based on C3. By integrating the global feature enhancement module to build the GC_3 feature extraction network, the feature extraction capability of the network is improved. Specifically:

[0042] Build the GC_3 network, which includes the TGC module, CBS module, BottleNeck module, and Concat module.

[0043] The TGC module is a feature enhancement module, which consists of two branches. One branch passes through a 3*3 convolution, then global average pooling, a 1*1 convolution, a ReLU activation layer and a 1*1 convolution; the other branch passes through a 5*5 convolution, then a 1*1 convolution, a Softmax layer, and then the result is added to the original input, and then passes through a 1*1 convolution, a ReLU activation layer and a 1*1 convolution. The output is then added. The specific network structure diagram is as follows: Figure 4 shown.

[0044] The CBS module includes a 1*1Conv layer, a BN layer, and a LeakyReLU activation layer in sequence to construct a feature map.

[0045] BottleNeck consists of two parts, 1*1 and 3*3 Conv. The 1*1 Conv is used to halve the number of channels in the feature map, and the 3*3 Conv doubles the number of channels. The two Convs increase the receptive field of the network and reduce the amount of calculation.

[0046] The GC_3 network mainly consists of three branches, such as Figure 3 As shown in the figure, the first branch contains one TGC module, one CBS module, and one BottleNeck module for deep feature extraction; the second branch contains a TGC module and a CBS module for simple channel adjustment and retains the original spatial information; the third branch only contains a CBS module; and then the three branches are connected through a Concat module.

[0047] After the picture is input into the feature extraction network, feature maps of different sizes are extracted. First, after passing through a 3*3 Conv, a feature map of size 160*160 and 64 channels is obtained, numbered P1; after passing through 3 GC_3 modules and a 3*3 Conv, a feature map of size 80*80 and 128 channels is obtained, numbered P2; after passing through 9 GC_3 modules and a 3*3 Conv, a feature map of size 40*40 and 256 channels is obtained, numbered P3; after passing through 9 GC_3 modules and a 3*3 Conv, a feature map of size 20*20 and 512 channels is obtained, numbered P4.

[0048] Step 4: Feature fusion is performed on the obtained feature map. The feature fusion network selects the FPN feature pyramid. Specifically, the feature map obtained after pooling the feature map numbered P4 through SPP is set to Q4. After upsampling Q4, the size of the feature map obtained is 40*40 and the number of channels is 256. It is fused with the feature map numbered P3 to obtain a new feature map, and the new feature map is numbered Q3; the size of the feature map obtained after Upsampling Q3 is 80*80 and the number of channels is 128. The feature map is fused with P2 to obtain a new feature map, and its number is set to Q2; the size of the feature map obtained after Upsampling Q2 is 160*160 and the number of channels is 64. The feature map is fused with the feature map of P1 to obtain a new feature map, and the number is set to Q1.

[0049] Step5: Step3 feature extraction obtains 4 layers of feature maps, P1, P2, P3, and P4. After Step4, we get Q1, Q2, Q3, and Q4 feature maps through the feature pyramid. As the feature layer goes deeper, the receptive field will increase, but some shallow information, such as more important texture information, will also be lost. Therefore, we select P1, P3, and P4 from the four layers of feature maps. It not only retains the important information of the shallow layer, but also takes into account the increase of the receptive field. The specific operations are:

[0050] Step 5.1: Add the CA attention mechanism after P1, and then fuse it with Q1 through skip connections to obtain C1.

[0051] Step 5.2: Downsample C1 to obtain C2.

[0052] Step 5.3: Add the CA attention mechanism after P3, and then fuse it with the downsampled result of C2 through skip connection to obtain C3.

[0053] Step 5.4: Add the CA attention mechanism after P4, and then fuse it with the downsampled result of C3 through skip connection to obtain C4.

[0054] Step 6: Add SK attention mechanism after C1, C3, and C4, and then input the result into the improved detection head New_Detect. Compared with the original detection head Detect of the YOLOv5 model, the improved detection head of the present invention changes the last two 3*3 convolutional layers of the original detection head Detect into one layer, which reduces the amount of calculation while ensuring accuracy. Detection by three detection heads includes both large target information and small target information.

[0055] Step 7: Use the data set of Step 2 to train the target detection network constructed in Step 3 to Step 6. The trained target detection network performs target detection on infrared images, inputs the image or video to be detected, and outputs the detection result.

[0056] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a target detection network of the target detection method suitable for infrared imaging scenes described in the above embodiment, which is used for target detection in infrared images.

[0057] The embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, implements the target detection network of the target detection method suitable for infrared imaging scenes described in the above embodiment. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

Claims

1. A target detection method suitable for infrared imaging scenes, characterized in that The following steps are involved: Step 1: Obtain the infrared image information collected by the infrared camera; Step 2: Build a vehicle and pedestrian dataset in infrared scenes; Step 3: Build an improved target detection network based on the YOLOv5 model. For the backbone network, build a feature extraction network based on C3 and build a feature extraction network by integrating the global attention mechanism. Step 4: The feature map obtained by the backbone network is subjected to feature fusion, and the feature fusion network is the feature pyramid FPN; Step 5: Fuse the original feature map of the backbone network with the feature map after the feature fusion network through skip connection to obtain fused feature maps of different scales; Step 6: After adding the SK attention mechanism to the fused feature maps of different scales, the maps are input into the detection head for detection and the target detection results are output; Step 7: Use the data set of Step 2 to train the target detection network constructed in Step 3 to Step 6, and use the trained target detection network to perform target detection on infrared images.

2. According to the target detection method suitable for infrared imaging scenes according to claim 1, it is characterized in that in Step 3, the feature extraction network built on the basis of C3 is called GC_3 network, which is specifically: The GC_3 network contains TGC module, CBS module, BottleNeck module and Concat module, among which: The TGC module is a feature enhancement module, which consists of two branches. One branch passes through a 3*3 convolution, then global average pooling, a 1*1 convolution, a ReLU activation layer and a 1*1 convolution; the other branch passes through a 5*5 convolution, then a 1*1 convolution, a Softmax layer, and then the result is added to the original input, and then passes through a 1*1 convolution, a ReLU activation layer and a 1*1 convolution. After that, the two branches are added to get the output. The CBS module includes 1*1Conv layer, BN layer, and LeakyReLU activation layer in sequence to construct feature maps; BottleNeck consists of two parts, 1*1 and 3*3 Conv. The 1*1 Conv is used to halve the number of feature map channels, and the 3*3 convolution doubles the number of channels. The two Convs are used to increase the network's receptive field and reduce the amount of calculation. The GC_3 network consists of three branches. The first branch contains one TGC module, one CBS module, and one BottleNeck module for deep feature extraction. The second branch contains the TGC module and the CBS module for channel adjustment and retaining the original spatial information. The third branch only contains the CBS module. The three branches are then connected through the Concat module. After the picture is input into the feature extraction network, feature maps of different sizes are extracted. After a 3*3 Conv, a feature map P1 with a size of 160*160 and 64 channels is obtained; after three GC_3 modules and a 3*3 Conv, a feature map P2 with a size of 80*80 and 128 channels is obtained; after nine GC_3 modules and a 3*3 Conv, a feature map P3 with a size of 40*40 and 256 channels is obtained; after nine GC_3 modules and a 3*3 Conv, a feature map P4 with a size of 20*20 and 512 channels is obtained.

3. According to the target detection method suitable for infrared imaging scenes according to claim 2, it is characterized in that the feature fusion network of Step 4 is specifically: The obtained feature map P4 is pooled by SPP to obtain the feature map Q4. After upsampling Q4, the feature map with a size of 40*40 and a number of channels of 256 is obtained, which is fused with the feature map P3 to obtain a new feature map Q3; the feature map with a size of 80*80 and a number of channels of 128 obtained after upsampling Q3 is fused with the feature map P2 to obtain a new feature map Q2; the feature map with a size of 160*160 and a number of channels of 64 obtained after upsampling Q2 is fused with the feature map P1 to obtain a new feature map Q1.

4. The target detection method suitable for infrared imaging scenes according to claim 3 is characterized in that Step 5 obtains fusion feature maps of different scales through jump connections: Step 5.1: Add the CA attention mechanism after the feature map P1, and then fuse it with the feature map Q1 through a skip connection to obtain the feature map C1; Step 5.2: Downsample the feature map C1 to obtain the feature map C2; Step 5.3: Add the CA attention mechanism after the feature map P3, and then fuse it with the down-sampled result of the feature map C2 through a skip connection to obtain the feature map C3; Step 5.3: Add the feature map CA attention mechanism after the feature map P4, and then fuse it with the down-sampled result of the feature map C3 through a jump connection to obtain the feature map C4.

5. The target detection method suitable for infrared imaging scenes according to claim 4 is characterized in that In Step 6, the feature maps C1, C3, and C4 are respectively input into the detection head for detection after passing through the SK attention mechanism. The detection head changes the two 3*3 convolutional layers in the detection head of YOLOv5 into one layer.

6. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the target detection network of the target detection method suitable for infrared imaging scenes described in any one of claims 1 to 5, which is used for target detection in infrared images.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, a target detection network of the target detection method suitable for infrared imaging scenes as described in any one of claims 1 to 5 is implemented.

Images