A target detection method, device and storage medium suitable for an infrared imaging scene

Abstract

A target detection method, device and storage medium suitable for an infrared imaging scene, an infrared camera is used to collect an infrared scene picture, the collected picture is labeled and then input into a network, and multi-layer feature maps are extracted through a newly constructed global feature extraction network; the extracted feature maps are input into a feature fusion network constructed by a feature pyramid for multi-scale feature fusion; then, in order to retain the micro target information in the shallow feature map, the original feature map and the fused feature map are fused again in a jump connection mode to obtain a final input; an output mode of three detection heads is adopted, which can ensure the detection of large targets and small targets. The present application ensures the feature extraction of the target in the infrared scene, and ensures the accuracy of the detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to recognition and detection of pedestrians and vehicles in an infrared image, and is a target detection method, device and storage medium suitable for an infrared imaging scene. BACKGROUND

[0002] An infrared image is obtained by detecting infrared radiation emitted by an object, and target detection can be performed under complex lighting environments. Compared with traditional visible light imaging technology, infrared imaging technology has the advantages of strong penetration, high measurement accuracy, less influence of weather, strong anti-external interference ability and the like. In the fields of automatic driving, security monitoring, military reconnaissance, auxiliary medical diagnosis and the like, the infrared imaging technology has wide applications. However, most of the current detection algorithms focus on the visible light field and are not suitable for the infrared imaging field. For example, the two-stage target detection algorithm represented by R-CNN and the one-stage detection algorithm represented by YOLO and SSD have good effects in the visible light scene, but in the infrared imaging scene, the demand for feature extraction cannot be met, and the problems of poor detection effect and low detection accuracy still exist. Due to the problems of large background interference, low image resolution, insufficient texture details and the like of the infrared image, the target feature extraction is difficult, and the problem of low detection accuracy still needs to be solved. SUMMARY

[0003] The problem to be solved by the application is that in the infrared scene, due to the problems of large similarity between the detection target and the image background, low image resolution, insufficient target texture details and the like, the accuracy of the infrared target detection network is low.

[0004] The technical scheme of the application is a target detection method suitable for an infrared imaging scene, comprising the following steps:

[0005] Step 1: acquiring infrared image information collected by an infrared camera;

[0006] Step 2: constructing a vehicle and pedestrian data set in an infrared scene;

[0007] Step 3: constructing an improved target detection network, based on a YOLOv5 model, a feature extraction network is built based on C3 for a backbone network, and the feature extraction network is built by integrating a global attention mechanism;

[0008] Step 4: performing feature fusion on the feature map obtained by the backbone network, and the feature fusion network is a feature pyramid network FPN;

[0009] Step 5: fusing the original feature map of the backbone network and the feature map after the feature fusion network to obtain a fusion feature map of different scales;

[0010] Step 6: After adding the SK attention mechanism to the fused feature maps of different scales, input into the detection head for detection, and output the target detection result;

[0011] Step 7: Train the target detection network constructed by Step 3-Step 6 using the data set of Step 2, and use the trained target detection network to realize target detection on the infrared image.

[0012] Further, in Step 3, the feature extraction network built based on C3 is called GC_3 network, which is specifically:

[0013] The GC_3 network includes a TGC module, a CBS module, a BottleNeck module and a Concat module.

[0014] The TGC module is composed of two branches. One branch passes through a 3*3 convolution, then a global average pooling, a 1*1 convolution, a ReLU activation layer and a 1*1 convolution; the other branch passes through a 5*5 convolution, then a 1*1 convolution, a Softmax layer, and then the obtained result is added to the original input, and then a 1*1 convolution, a ReLU activation layer and a 1*1 convolution are added, and then the two branches are added to obtain the output.

[0015] The CBS module includes a 1*1 Conv layer, a BN layer and a LeakyReLU activation layer in sequence, which is used to construct a feature map.

[0016] The BottleNeck includes two parts, which are 1*1 and 3*3 Conv, respectively. The 1*1 Conv is used to reduce the number of feature map channels by half, and the 3*3 convolution doubles the number of channels. Through the two Convs, the receptive field of the network is increased, and the calculation amount is reduced.

[0017] The GC_3 network includes three branches. The first branch includes a TGC module, a CBS module and a BottleNeck module, which are used for deep feature extraction; the second branch includes a TGC module and a CBS module, which are used for channel adjustment and preservation of original spatial information; the third branch only includes a CBS module; then the three branches are connected through a Concat module.

[0018] The picture input feature extraction network extracts feature maps of different sizes, and first obtains a feature map P1 of 160*160 and 64 channels after 3*3 Conv; then obtains a feature map P2 of 80*80 and 128 channels after 3 GC_3 modules and a 3*3 Conv; then obtains a feature map P3 of 40*40 and 256 channels after 9 GC_3 modules and a 3*3 Conv; and then obtains a feature map P4 of 20*20 and 512 channels after 9 GC_3 modules and a 3*3 Conv.

[0019] Further, the feature fusion network of Step4 is specifically:

[0020] The feature map P4 obtained is pooled by SPP to obtain a feature map Q4, and the feature map obtained after upsampling Q4 is 40*40 and 256 channels, which is fused with the feature map P3 to obtain a new feature map Q3; the feature map obtained after upsampling Q3 is 80*80 and 128 channels, which is fused with the feature map P2 to obtain a new feature map Q2; the feature map obtained after upsampling Q2 is 160*160 and 64 channels, which is fused with the feature map P1 to obtain a new feature map Q1.

[0021] Further, the different scale fusion feature maps obtained by Step5 through skip connection are specifically:

[0022] Step5.1: CA attention mechanism is added after the feature map P1, and then fused with the feature map Q1 through skip connection to obtain a feature map C1;

[0023] Step5.2: the feature map C1 is down-sampled to obtain a feature map C2;

[0024] Step5.3: CA attention mechanism is added after the feature map P3, and then fused with the down-sampled result of the feature map C2 through skip connection to obtain a feature map C3;

[0025] Step5.3: CA attention mechanism is added after the feature map P4, and then fused with the down-sampled result of the feature map C3 through skip connection to obtain a feature map C4.

[0026] Further, in Step6, the feature maps C1, C3 and C4 are respectively input into the detection head after SK attention mechanism for detection, and the detection head changes two 3*3 convolution layers in the detection head of YOLOv5 into one layer.

[0027] The application further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor, and a target detection network of the target detection method suitable for an infrared imaging scene is realized for target detection of an infrared image.

[0028] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is executed to realize the target detection network of the target detection method suitable for the infrared imaging scene.

[0029] Compared with the prior art, the application has the following advantages:

[0030] Firstly, the application constructs a TGC feature enhancement module, and a GC_3 feature extraction network is constructed through the module, compared with a traditional feature extraction network, the method adopts the newly constructed TGC module to model the global, so that the model can extract the relationship of the global context, and ensure that the global context information is considered, meanwhile, the TGC processes multi-scale information through a multi-branch structure, to help the network obtain more required feature information, help the network capture more important detail information, generate more rich feature representation, avoid feature loss, and improve the robustness of the network, and through aggregation of different features of multiple branches, the processing capacity of the network bottleneck layer is enhanced;

[0031] Secondly, the application adopts multiple jump connections, and the original feature map containing more rich information and the feature map containing deeper features after feature fusion are fused again, so that the fused feature map contains not only the shallow position information in the original feature map, but also deep semantic information, and the ability of the network to process feature information from a global level is improved;

[0032] Thirdly, the application solves the problems of difficult feature extraction and poor detection result of the infrared image to a great extent through the improved feature extraction network and the multi-level feature fusion network, and in combination with multiple attention mechanisms, the accuracy of the application is improved by 6.7% compared with the existing detection network through test of the Aireal infrared open source dataset. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The figure is a flowchart of the application.

[0034] Figure 2 The figure is a structure diagram of the improved detection network of the application.

[0035] Figure 3 The figure is a network structure diagram of the feature extraction network GC_3 of the application.

[0036] Figure 4 This is a schematic diagram of the TGC module structure in the GC_3 network of the present invention. Detailed Implementation

[0037] This invention proposes a target detection and recognition method suitable for infrared imaging scenarios. It solves the problems of difficult target feature extraction and low detection accuracy in infrared scenes. This invention is highly practical, applicable to various scenarios such as autonomous driving and surveillance, and can be deployed on a large scale.

[0038] like Figure 1 As shown, the specific implementation process of the present invention is as follows.

[0039] Step 1: Obtain infrared image information captured by the vehicle-mounted infrared camera.

[0040] Step 2: Construct an infrared pedestrian and vehicle dataset, label pedestrian and vehicle bounding boxes and indicate their categories, and divide the classified images into training and test sets in a 9:1 ratio.

[0041] Step 3: Construct an improved object detection network. Based on the YOLOv5 model, a feature extraction network is built on the C3 backbone network. A GC_3 feature extraction network is constructed by incorporating a global feature enhancement module to improve the network's feature extraction capabilities. Specifically:

[0042] Set up the GC_3 network, which includes the TGC module, CBS module, BottleNeck module, and Concat module.

[0043] The TGC module is a feature enhancement module, consisting of two branches. One branch uses a 3x3 convolution, followed by Global Average Pooling, a 1x1 convolution, a ReLU activation layer, and another 1x1 convolution. The other branch uses a 5x5 convolution, followed by a 1x1 convolution, a Softmax layer, and then adds the result to the original input. This is followed by another 1x1 convolution, a ReLU activation layer, and another 1x1 convolution. Finally, the two branches are summed to obtain the output. The specific network structure diagram is shown below. Figure 4 As shown.

[0044] The CBS module consists of a 1*1 Conv layer, a BN layer, and a LeakyReLU activation layer, which are used to construct feature maps.

[0045] BottleNeck contains two parts, 1*1, 3*3 Conv, respectively, wherein 1*1 Conv is used to reduce the number of feature map channels by half, and 3*3 convolution doubles the number of channels. Through two Convs, the receptive field of the network is increased, and the calculation amount is reduced.

[0046] The GC_3 network mainly contains three branches, as shown in Figure 3 The first branch contains one TGC module, one CBS module and one BottleNeck module for deep feature extraction; the second branch contains a TGC module and a CBS module for simple channel adjustment and preservation of original spatial information; the third branch contains only a CBS module; then the three branches are connected through a Concat module.

[0047] After the picture input feature extraction network, different size feature maps are extracted, first through a 3*3 Conv to obtain a 160*160 feature map with 64 channels, numbered P1; then through three GC_3 modules and a 3*3 Conv, a 80*80 feature map with 128 channels is obtained, numbered P2; then through nine GC_3 modules and a 3*3 Conv, a 40*40 feature map with 256 channels is obtained, numbered P3; then through nine GC_3 modules and a 3*3 Conv, a 20*20 feature map with 512 channels is obtained, numbered P4.

[0048] Step 4: The obtained feature maps are fused, and the FPN feature pyramid is selected for feature fusion network. Specifically, the feature map numbered P4 is set as Q4 after SPP pooling, and the size of the feature map obtained after up-sampling Upsample of Q4 is 40*40 and the number of channels is 256. The feature map is fused with the feature map numbered P3 to obtain a new feature map, and the new feature map is numbered Q3. The size of the feature map obtained after Upsample of Q3 is 80*80 and the number of channels is 128. The feature map is fused with P2 to obtain a new feature map, and the new feature map is numbered Q2. The size of the feature map obtained after Upsample of Q2 is 160*160 and the number of channels is 64. The feature map is fused with P1 feature to obtain a new feature map, and the new feature map is numbered Q1.

[0049] Step5: Step3 feature extraction obtains P1, P2, P3, P4, a total of 4 layers of feature maps, after Step4, Q1, Q2, Q3, Q4 feature maps are obtained after the feature pyramid. With the deepening of the feature layer, the receptive field will increase, but some shallow information, such as important texture information, will also be lost. Therefore, we select P1, P3 and P4 from P1, P2, P3 and P4. Both important information in the shallow layer and the increase of the receptive field are taken into account. The specific operation is as follows:

[0050] Step5.1: CA attention mechanism is added after P1, and then fused with Q1 through jump connection to obtain C1.

[0051] Step5.2: C1 is down-sampled to obtain C2.

[0052] Step5.3: CA attention mechanism is added after P3, and then fused with the result of C2 after down-sampling to obtain C3.

[0053] Step5.4: CA attention mechanism is added after P4, and then fused with the result of C3 after down-sampling to obtain C4.

[0054] Step6: SK attention mechanism is added after C1, C3 and C4, and then the result is input into the improved detection head New_Detect. Compared with the original detection head Detect of YOLOv5 model, the improved detection head changes the last two 3*3 convolution layers of the original detection head Detect into one layer, which reduces the amount of calculation while ensuring the accuracy. Through three detection heads, both large target information and small target information are contained.

[0055] Step7: The target detection network constructed by Step3-Step6 is trained by using the data set of Step2, and the trained target detection network is used for target detection of infrared images. The input is a picture or video to be detected, and the output is a detection result.

[0056] The embodiment of the application also provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor, and the target detection network of the target detection method suitable for the infrared imaging scene is realized, which is used for target detection of infrared images.

[0057] The embodiment of the present application also provides a computer readable storage medium storing a computer program, wherein the computer program implements the target detection network of the target detection method suitable for an infrared imaging scene when executed by a processor of a computer. The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The program code can be executed completely on a machine, partially on a machine, partially on a machine as a separate software package and partially on a remote machine, or completely on a remote machine or server.

Claims

1. A target detection method suitable for infrared imaging scenarios, characterized in that: Includes the following steps: Step 1: Acquire infrared image information captured by the infrared camera; Step 2: Construct a dataset of vehicles and pedestrians in an infrared scene; Step 3: Construct an improved object detection network. Based on the YOLOv5 model, the feature extraction network is built on C3 as the backbone network. The feature extraction network is constructed by incorporating a global attention mechanism. Step 4: Perform feature fusion on the feature maps obtained from the backbone network. The feature fusion network is the Feature Pyramid Network (FPN). Step 5: Fuse the original feature map of the backbone network with the feature map after passing through the feature fusion network through skip connections to obtain fused feature maps of different scales; Step 6: After fusing feature maps at different scales, add the SK attention mechanism, input them into the detection head for detection, and output the target detection results; Step 7: Use the dataset from Step 2 to train the target detection network constructed in Steps 3-6. The trained target detection network will then be used to detect targets in the infrared image. In Step 3, the feature extraction network built on C3 is called the GC_3 network, specifically: The GC_3 network includes the TGC module, CBS module, BottleNeck module, and Concat module, among which: The TGC module is a feature enhancement module consisting of two branches. One branch uses a 3x3 convolution followed by global average pooling, a 1x1 convolution, a ReLU activation layer, and another 1x1 convolution. The other branch uses a 5x5 convolution followed by a 1x1 convolution, a softmax layer, and then adds the result to the original input. This is then followed by another 1x1 convolution, a ReLU activation layer, and another 1x1 convolution. Finally, the two branches are summed to obtain the output. The CBS module consists of a 1*1 Conv layer, a BN layer, and a LeakyReLU activation layer, which are used to construct feature maps. BottleNeck consists of two parts: a 1*1 convolution and a 3*3 convolution. The 1*1 convolution halves the number of channels in the feature map, while the 3*3 convolution doubles the number of channels. The two convolutions increase the receptive field of the network and reduce the computational cost. The GC_3 network consists of three branches. The first branch contains one TGC module, one CBS module, and one BottleNeck module for deep feature extraction. The second branch contains both TGC and CBS modules for channel adjustment and preservation of original spatial information. The third branch contains only the CBS module. These three branches are then connected using a Concat module. After the image is input into the feature extraction network, feature maps of different sizes are extracted. First, after passing through a 3*3 Conv, a feature map P1 with a size of 160*160 and 64 channels is obtained; then after passing through 3 GC_3 modules and a 3*3 Conv, a feature map P2 with a size of 80*80 and 128 channels is obtained; then after passing through 9 GC_3 modules and a 3*3 Conv, a feature map P3 with a size of 40*40 and 256 channels is obtained; finally, after passing through 9 GC_3 modules and a 3*3 Conv, a feature map P4 with a size of 20*20 and 512 channels is obtained.

2. The target detection method suitable for infrared imaging scenarios according to claim 1, characterized in that the feature fusion network in Step 4 is specifically as follows: The obtained feature map P4 is pooled using SPP to obtain feature map Q4. After upsampling Q4, a feature map of size 40*40 and number of channels 256 is obtained. This feature map is then fused with feature map P3 to obtain a new feature map Q3. After upsampling Q3, a feature map of size 80*80 and number of channels 128 is obtained. This feature map is then fused with feature map P2 to obtain a new feature map Q2. After upsampling Q2, a feature map of size 160*160 and number of channels 64 is obtained. This feature map is then fused with feature map P1 to obtain a new feature map Q1.

3. The target detection method suitable for infrared imaging scenarios according to claim 2, characterized in that: Step 5 involves obtaining fused feature maps at different scales through skip connections, specifically as follows: Step 5.1: Add a CA attention mechanism after feature map P1, and then fuse it with feature map Q1 through a skip connection to obtain feature map C1; Step 5.2: Downsample feature map C1 to obtain feature map C2; Step 5.3: Add a CA attention mechanism after feature map P3, and then fuse it with the downsampled result of feature map C2 through a skip connection to obtain feature map C3; Step 5.3: Add a feature map CA attention mechanism after feature map P4, and then fuse it with the downsampled result of feature map C3 through skip connections to obtain feature map C4.

4. A target detection method suitable for infrared imaging scenarios according to claim 3, characterized in that: In Step 6, feature maps C1, C3, and C4 are fed into the detection head after passing through the SK attention mechanism. The detection head replaces the two 3*3 convolutional layers in the YOLOv5 detection head with a single layer.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the target detection network of the target detection method suitable for infrared imaging scenarios according to any one of claims 1-4, for target detection in infrared images.

6. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements a target detection network for a target detection method suitable for infrared imaging scenarios as described in any one of claims 1-4.

Images