An infrared target detection method applied to complex environment

Abstract

The application belongs to the field of infrared target detection, and particularly relates to an infrared target detection method applied to a complex environment, comprising the following steps: acquiring an infrared image to be detected, and pre-processing the infrared image; extracting different scale features of the infrared image by using a backbone feature extraction network; performing strengthening fusion processing on the features of different scales by using a neck strengthening feature extraction network to obtain a fused feature map; inputting the fused effective feature map into a prediction output network to obtain a detection result of the target; and effectively improving the infrared target detection precision, and better detecting the infrared target which is easily shielded in a complex scene, while significantly reducing the parameter quantity and meeting the real-time detection requirement.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of infrared target detection, and particularly relates to an infrared target detection method applied to a complex environment. BACKGROUND

[0002] Infrared images are obtained through thermal radiation, and have the outstanding features of long target detection distance, strong concealment, and availability in both day and night. With the expansion of the distance imaging range, the demand for intelligent target detection methods in infrared images is becoming more and more urgent. Traditional infrared image target detection methods include threshold-based methods and edge detection-based methods, etc., but such methods are only suitable for detection in a single scene. Due to the complexity of the real environment and the weak infrared target features, it is difficult to accurately detect the target, which leads to the difficulty of the detection model in extracting important features for some targets hidden by obstacles, and poor practicability. The detection method based on convolutional neural network can automatically learn features from input data, and has robustness to changes in complex environments and stronger adaptability.

[0003] At present, existing infrared target detection methods include the patent application number CN202210207336.X, which is a method for infrared target detection in a complex scene. The method includes Mosaic data enhancement on the input infrared image; optimization and improvement of the CSPDarknet53 structure of the feature extraction network, and addition of an attention mechanism ECA module in the feature extraction network; slice operation on the input image using the Focus structure, followed by multiple convolution processing, and then feature information extraction using the optimized CSPDarknet53 feature extraction network to obtain feature maps of different scales, and addition of an SPP module after the feature extraction network to solve the problem of precision decline caused by target scale change; the smallest feature map is passed through a feature pyramid network structure and a path aggregation network structure to fuse the strong semantic feature information of the high layer and the strong positioning feature of the low layer, and finally obtain detection layers of different scales with strong semantic features and strong positioning features through the combination of the two network structures; Varifocal Loss is used as the loss function of the confidence and class probability of the detected object to realize multi-scale detection and obtain different prediction boxes. The patent extracts features from the input infrared image through the improved backbone feature extraction network, combines the feature pyramid network structure and the path aggregation network structure to realize the fusion of feature information of different scales, optimizes the loss function of the network, finally predicts the feature maps of different scales, and uses the non-maximum suppression based on Distance-IoU (DIoU) to improve the detection of densely occluded objects, which can be widely applied in the fields of automatic driving and night security.

[0004] However, the above method has the following problems: 1. The feature extraction module CSPDarknet53 has too many parameters in the feature extraction process, and the feature map is redundant. 2. The multi-scale feature fusion needs to be strengthened, and the anti-background interference ability is weak. SUMMARY

[0005] To solve the problems existing in the prior art, the present application provides an infrared target detection method applied to a complex environment, which comprises the following steps: acquiring an infrared image to be detected, and preprocessing the infrared image; inputting the preprocessed infrared image into a trained infrared target detection model to obtain a detection result; wherein the infrared target detection model comprises a backbone feature extraction network, a neck enhanced feature extraction network, and a prediction output feature layer.

[0006] The process of training the infrared target detection model comprises:

[0007] S1: acquiring a training data set, wherein the training data set comprises infrared images and class labels corresponding to the infrared images;

[0008] S2: preprocessing the infrared images in the training data set and inputting the preprocessed infrared images into the infrared target detection model for training;

[0009] S3: extracting different scale features of the infrared image by using the backbone feature extraction network;

[0010] S4: enhancing and fusing the features of different scales by using the neck enhanced feature extraction network to obtain a fused feature map;

[0011] S5: inputting the fused feature map into the prediction output feature layer to obtain a detection result of the target;

[0012] S6: calculating a loss function of the model according to the target detection result, and constantly adjusting the model parameters, using a performance evaluation index to determine the accuracy of the target detection result, and when the accuracy of the target detection result meets the requirements, the training of the model is completed.

[0013] Preferably, the backbone feature extraction network comprises a GSeConv module, a C3Ghost module and a SPPF module, wherein the GSeConv module is used to extract shallow features of the infrared image, the C3Ghost module is used to reduce redundant information of the shallow features, and the SPPF module is used to increase the receptive field of the network to obtain context information after removing redundant information.

[0014] Further, the GSeConv module extracts the shallow features of the infrared image, including: using a 1*1 size convolution to compress the channels of the infrared image, wherein the number of convolution kernels of the convolution layer is half of the number of input image channels; using a 3*3 size layer-by-layer convolution to reconstruct the features of the 1*1 size convolution output, to obtain a mixed feature map; splitting the mixed feature map into two groups of feature maps; superimposing the first group of feature maps and the condensed features generated by point-by-point convolution in the channel direction; and splicing the superimposed feature map and the second group of feature maps to obtain an output result.

[0015] Preferably, the neck strengthening feature extraction network EPANet is used to strengthen and fuse features of different scales, including: passing the 32 times down-sampling feature map extracted by the backbone network through the SPPF module to obtain an output feature map with a size of 20*20; transforming the size of the 32 times down-sampling feature map through a 1*1 convolution module and an up-sampling module, splicing the transformed feature map and the 16 times down-sampling feature map of the backbone network to obtain a feature map A; inputting the feature map A into the C3GS module and the up-sampling module to further extract features and superimpose the feature map obtained by the 8 times down-sampling of the backbone network to obtain a feature map B, realizing a bottom-up fusion process; inputting the feature map B into the C3GS module and the down-sampling module for processing to obtain a 40*40 fusion feature map and an 80*80 fusion feature map.

[0016] Preferably, the feature extraction module C3GS module of the neck network can reduce the parameter amount without losing accuracy. The module structure feature is that after channel splitting, there are two branches, one branch does not do any operation, and the other branch first extracts features through the GSeConv module, then uses the SimAM attention mechanism to weight the corresponding feature information, thereby highlighting the effective feature details in the image, and shuffles the superimposed feature map to promote information interaction between channels and enhance the learning ability of the network. The neck multi-scale feature fusion network realizes the bottom-up and top-down EPANet enhanced information fusion network through up-sampling and down-sampling features. In order to achieve better fusion effect, the nodes with smaller contribution to the network are removed, the depth of the network is reduced, and the neck is made lighter; then a fusion edge is added between the original shallow network and the bottom layer output node of the neck to fuse higher level features, shorten the transmission path of the context information, and extract more rich semantic information features.

[0017] Preferably, the expression of the model loss function is:

[0018]

[0019]

[0020] wherein, IOU represents the intersection over union of the predicted frame and the real target frame, and p 2 (b,b gt ) represents the distance between the center points of the predicted frame and the real frame, p represents the Euclidean distance between the center points of the two frames, b represents the coordinate of the predicted center point, and b gt represents the coordinate of the center point of the real target frame, c represents the diagonal distance of the smallest rectangular region containing the two boundary frames, alpha is a parameter for balancing the scale, v is used to measure the consistency of the aspect ratio, w gt and h gt represent the width and height of the real target frame, and w and h represent the width and height of the predicted frame.

[0021] Preferably, the calculation formula of the performance evaluation index is:

[0022]

[0023]

[0024] wherein, precision represents the precision, tp represents the number of detection frames with the intersection over union of the predicted value and the real value greater than 0.5, fp represents the number of detection frames with the intersection over union of the predicted value and the real value less than or equal to 0.5, recall represents the recall, and fn represents the number of real values that are not detected.

[0025] Beneficial effects:

[0026] The present application effectively improves the infrared target detection precision, and the detection effect of the infrared target that is easily shielded in a complex scene is better, while the parameter amount is significantly reduced, the real-time detection requirement is met, and the deployment of edge devices is friendly. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is the flow chart of the infrared target detection algorithm in the embodiment of the present application;

[0028] Figure 2 It is the structure diagram of the GSeConv module in the embodiment of the present application;

[0029] Figure 3 It is the structure diagram of the C3Ghost module in the embodiment of the present application;

[0030] Figure 4 It is the structure diagram of the C3GS module in the embodiment of the present application;

[0031] Figure 5 It is the structure diagram of the SimAM module in the embodiment of the present application;

[0032] Figure 6 It is the network structure diagram of the EPANet in the embodiment of the present application;

[0033] Figure 7 Figure 1 is a schematic diagram of an infrared detection model according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0035] A specific embodiment of an infrared target detection method applied to a complex environment, the method comprising: acquiring an infrared image to be detected, and preprocessing the infrared image; inputting the preprocessed infrared image into a trained infrared target detection model to obtain a detection result; wherein the infrared target detection model comprises a backbone feature extraction network, a neck enhanced feature extraction network, and a prediction output feature layer. Specifically, as shown in FIG. 1, a given infrared image is first extracted by a shallow backbone network to obtain basic contour, texture and other fine-grained feature information; the backbone network mainly extracts features by GSeConv modules and C3Ghost modules; different scale features extracted by the backbone network are fused by the neck EPANet network for multi-scale information fusion, and the extraction of target features is further enhanced; the main feature extraction module of the neck network is a C3GS module; the prediction output feature layer utilizes three effective feature layers processed by the fusion network, identifies and predicts the target category, and regresses the position of the detection frame, so as to obtain an output result. Figure 1

[0036] In the present embodiment, a method for training an infrared target detection model is provided, the method comprising:

[0037] S1: acquiring a training data set, wherein the training data set comprises infrared images and category labels corresponding to the infrared images;

[0038] S2: preprocessing the infrared images in the training data set, and inputting the preprocessed infrared images into an infrared target detection model for training;

[0039] S3: extracting different scale features of the infrared images by using a backbone feature extraction network;

[0040] S4: enhancing and fusing the different scale features by using a neck enhanced feature extraction network to obtain a fused feature map;

[0041] S5: inputting the fused feature map into a prediction output feature layer to obtain a detection result of the target; ​

[0042] S6: Calculate the loss function of the model according to the target detection result, and continuously adjust the model parameters, use the performance evaluation index to determine the accuracy of the target detection result, when the accuracy of the target detection result meets the requirements, complete the training of the model.

[0043] As shown in Figure 7 The backbone feature extraction network includes a GSeConv module, a C3Ghost module and an SPPF module, wherein the GSeConv module is used to extract shallow features of the infrared image, the C3Ghost module is used to reduce redundant information of the shallow features, and the SPPF module is used to increase the receptive field of the network to obtain context information after removing redundant information.

[0044] Specifically, the shallow layer contains basic contour, texture and other fine-grained feature information, and insufficient extraction of feature information in the shallow network will cause the detected target to lose part of the information and blur the expression of global features, which will directly affect the feature extraction quality of the deep network, thereby reducing the detection accuracy of the model. As shown in Figure 2 The GSeConv module is the core feature extraction module of the shallow part of the backbone, which can effectively suppress irrelevant information interference, reduce the loss of shallow features, and enhance the expression of target feature information. The specific operation of this module is as follows: first, a 1x1 convolution is used to compress the channels of the input feature map, wherein the number of convolution kernels is set to half of the number of output channels, then a 3x3 size layer-by-layer convolution is used to reconstruct the features with less parameter amount. Based on this, the mixed feature maps obtained are split into two groups of feature maps, one group of condensed features generated by point-by-point convolution is stacked in the channel direction, and the other group of features is directly mapped to the next layer and spliced with the stacked features for convolution operation, thereby obtaining the output result. As shown in Figure 3 The C3Ghost uses the CSP architecture, which solves the problem of repeated gradient information injection network, makes the gradient flow into different paths, and makes the propagation of gradient information have correlation difference, the purpose is to reduce the inference cost while improving the accuracy.

[0045] The neck strengthening feature extraction network is used for strengthening fusion of features of different scales. The effective feature layer obtained by the backbone network is fused with the features of the neck network after upsampling and downsampling, so as to realize the fusion process from bottom to top and from top to bottom. In order to achieve better fusion effect, the nodes with less contribution to the network are removed, the depth of the network is reduced, and the neck is made lighter; then a fusion edge is added between the original shallow network and the output nodes at the bottom of the neck, so as to fuse the features of higher levels, shorten the transmission path of the context information, and extract more rich semantic information features. The feature extraction module C3GS module of the neck network can reduce the parameter amount without losing accuracy. The module structure is characterized in that the feature maps are subjected to channel splitting processing, a group of feature maps after splitting is input into the GSeConv module to extract deep features; the SimAM attention mechanism is used to weight process the deep feature information; and the stacked output feature maps are shuffled by channel to promote information interaction and enhance the learning ability of the network.

[0046] The GSeConv module extracts shallow features of the infrared image, including: using a 1x1 size convolution to compress the channels of the infrared image, wherein the number of convolution kernels of the convolution layer is half of the number of input image channels; using a 3x3 size layer-by-layer convolution to reconstruct features of the feature maps output by the 1x1 size convolution, to obtain mixed feature maps; the mixed feature maps are split by channel to obtain two groups of feature maps; the first group of feature maps and the condensed features generated by point-by-point convolution are stacked by channel; the stacked feature maps and the second group of feature maps are spliced to obtain an output result.

[0047] The C3Ghost module includes a residual network branch and a convolution branch, wherein the residual network branch is composed of multiple residual structures; the input information is input into the residual network branch and the convolution branch, wherein the residual network branch is used to extract deep features of the input information and increase the gradient value of the back propagation between layers, and the convolution branch is used to directly extract features of the input information; the feature maps extracted by the two branches are stacked by channel to obtain output feature maps.

[0048] In the embodiment, the feature extraction module C3GS module of the neck network can reduce the parameter amount without losing accuracy. As shown in Figure 4 , the module structure is characterized in that after channel splitting, two branches are formed, one branch is not operated, and the other branch first extracts features by the GSeConv module, then uses the SimAM attention mechanism to weight the corresponding feature information, so as to highlight the effective feature details in the image, and shuffles the stacked feature maps to promote information interaction between channels and enhance the learning ability of the network, wherein the structure of the SimAM attention mechanism is as shown in Figure 5 . As shown in Figure 6As shown, the neck multi-scale feature fusion network realizes the bottom-up and top-down EPANet enhanced information fusion network through up-sampling and down-sampling features. In order to achieve better fusion effect, the nodes with less contribution to the network are removed, the depth of the network is reduced, and the neck is made lighter; then a fusion edge is added between the original shallow network and the output nodes at the bottom layer of the neck, to fuse the features at a higher level, shorten the transmission path of the context information, and thus extract richer semantic information features.

[0049] The neck enhanced feature extraction network EPANet is adopted to perform enhanced fusion processing on features of different scales, including: the 32 times down-sampling feature map extracted by the backbone network is input into the SPPF module to obtain an output feature map with a size of 20x20; the 32 times down-sampling feature map is transformed in size by a 1x1 convolution module and an up-sampling module, and the transformed feature map is spliced with the 16 times down-sampling feature map of the backbone network to obtain a feature map A; the feature map A is input into the C3GS module and the up-sampling module to further extract features and stack the features obtained by the 8 times down-sampling of the backbone network to obtain a feature map B, realizing a bottom-up fusion process; the feature map B is input into the C3GS module and the down-sampling module for processing to obtain a 40x40 fusion feature map and an 80x80 fusion feature map.

[0050] The C3GS module includes a convolution branch and a special processing branch. The convolution branch directly extracts input channel information, and the special processing branch performs channel splitting processing on the feature map, inputs the split feature map into the GSeConv module to extract deep features; the SimAM attention mechanism is adopted to weight process the deep feature information; and the stacked output feature map is shuffled by channel to promote information interaction. The feature maps obtained by the two branches are stacked and output.

[0051] The post-processing stage of the prediction output end of the model includes target class confirmation and regression positioning. The regression positioning loss function of the detection frame position is CIOU, which not only considers the overlapping area and center point distance between the frame prediction frame and the real frame, but also introduces two target boundary frame influence factors.

[0052] Specifically, the process of processing the fusion feature map by the prediction output feature layer includes: inputting three fusion feature maps output by the neck enhanced feature extraction network EPANet into three corresponding prediction output feature layers respectively; assigning three scales of preset prior boxes in each prediction feature layer; taking a detection head for predicting small targets as an example, dividing the 80x80 fusion feature map into 80x80 grids, and each grid corresponds to three preset prior boxes; using a predictor to slide on the divided fusion feature map, and when the predictor slides on a specific grid, the confidence parameters and regression parameters of each prior box are predicted; using a CIOU loss function to regress and fine-tune the predicted box, decode all category information and coordinate information of the position of the predicted box, so as to realize positioning of the target box; removing redundant predicted boxes through non-maximum suppression, retaining the optimal predicted box, and obtaining the final detection result.

[0053] The expression of the model loss function is:

[0054]

[0055]

[0056] wherein, IOU represents the intersection over union of the predicted box and the real target box, ρ 2 (b,b gt ) represents the distance between the predicted box and the real box center point, ρ represents the Euclidean distance between the two box center points, b represents the predicted center point coordinates, b gt represents the real target box center point coordinates, c represents the distance of the diagonal of the smallest rectangular region containing the two boundary boxes, α is a parameter for balancing the scale, v is used to measure the consistency of the height-width ratio, w gt and h gt represent the width and height of the real target box, and w and h represent the width and height of the predicted box.

[0057] The commonly used performance indicators for target detection tasks are:

[0058] Precision (P): the ratio of positive samples in correctly identified positive and negative samples.

[0059]

[0060] Recall (R): the proportion of correctly identified positive samples in all positive samples in the test set.

[0061]

[0062] Mean Average Precision (mAP): the average of the average correct rates of each category.

[0063] Wherein, tp is the number of detection boxes whose prediction value and real value intersection ratio is greater than 0.5; fp is the number of detection boxes whose prediction value and real value intersection ratio is less than or equal to 0.5; fn is the number of real values that are not detected.

[0064] Simulation implementation conditions: environment configuration is based on Pytorch 1.12.0, CUDA 11.7, python3.7. The model is trained on an NVIDIA GeForce RTX 3060 GPU. In the training stage, the optimizer weight decay is set to 0.005, the SGD momentum is 0.9, and the model is trained for 100 iterations.

[0065] Compared with the most advanced infrared target detection model, the detection accuracy of the present application is improved by 2 points, and the detection effect of the infrared small target hidden in the complex scene is better.

[0066] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made to the present application within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An infrared target detection method applied to a complex environment, characterized in that, The method comprises the following steps: acquiring an infrared image to be detected, and preprocessing the infrared image; inputting the preprocessed infrared image into a trained infrared target detection model to obtain a detection result; wherein the infrared target detection model comprises a backbone feature extraction network, a neck enhanced feature extraction network, and a prediction output feature layer; the process of training the infrared target detection model comprises the following steps: S1: acquiring a training data set, wherein the training data set comprises infrared images and category labels corresponding to the infrared images; S2: preprocessing the infrared images in the training data set, and inputting the preprocessed infrared images into the infrared target detection model for training; S3: extracting different scale features of the infrared image by using the backbone feature extraction network; the backbone feature extraction network comprises a GSeConv module, a C3Ghost module, and a SPPF module, wherein the GSeConv module is used to extract shallow features of the infrared image, the C3Ghost module is used to reduce redundant information of the shallow features, and the SPPF module is used to increase a receptive field of the network to obtain context information after removing the redundant information; the GSeConv module extracts the shallow features of the infrared image, which comprises the following steps: compressing channels of the infrared image by using a 1×1 size convolution, wherein the number of convolution kernels of the convolution is half of the number of input image channels; reconstructing features of the convolution output of the 1×1 size convolution by using a 3×3 size convolution layer by layer to obtain a mixed feature map; splitting the mixed feature map in the channel direction to obtain two groups of feature maps; superimposing the first group of feature maps and condensed features generated by point-by-point convolution in the channel direction; and splicing the superimposed feature map and the second group of feature maps to obtain an output result; S4: enhancing the features of different scales by using the neck enhanced feature extraction network to obtain a fusion feature map; S5: inputting the fusion feature map into the prediction output feature layer to obtain a detection result of the target; S6: calculating a loss function of the model according to the target detection result, and constantly adjusting model parameters, using a performance evaluation index to determine the accuracy of the target detection result, and completing the training of the model when the accuracy of the target detection result meets the requirements.

2. The method for infrared target detection in complex environment according to claim 1, characterized in that, The C3Ghost module comprises a residual network branch and a convolution branch, wherein the residual network branch is composed of a plurality of residual structures; the input information is input into the residual network branch and the convolution branch, wherein the residual network branch is used to extract deep features of the input information and increase gradient values of back propagation between layers, and the convolution branch is used to directly extract features of the input information; the feature maps extracted by the two branches are superimposed in the channel direction to obtain an output feature map.

3. The method for infrared target detection in complex environment according to claim 1, characterized in that, The neck strengthening feature extraction network EPANet is used to strengthen and fuse features of different scales, including: the 32 times down-sampling feature map extracted by the backbone network is input into the SPPF module to obtain an output feature map with a size of 20*20; the 32 times down-sampling feature map is transformed in size by a 1*1 convolution module and an up-sampling module, and the transformed feature map is spliced with the 16 times down-sampling feature map of the backbone network to obtain a feature map A; the feature map A is input into the C3GS module and the up-sampling module to further extract features and stack the features with the feature map obtained by 8 times down-sampling of the backbone network to obtain a feature map B; the feature map B is input into the C3GS module and the down-sampling module for processing to obtain a 40*40 fusion feature map and an 80*80 fusion feature map; The C3GS module includes a convolution branch and a special processing branch, the convolution branch directly extracts input channel information, the special processing branch performs channel splitting processing on the feature map, and a group of feature maps after splitting are input into the GSeConv module to extract deep features; the SimAM attention mechanism is used to weight process the deep feature information; and the stacked output feature map is shuffled by channel to promote information interaction; the feature maps obtained by the two branches are stacked and output.

4. The method for infrared target detection in complex environment according to claim 1, characterized in that, The process of processing the fusion feature map by the prediction output feature layer includes: inputting the three fusion feature maps output by the neck strengthening feature extraction network EPANet into three corresponding prediction output feature layers; three scales of preset prior boxes are allocated in each prediction feature layer; in the small target detection process, the 80*80 fusion feature map is divided into 80*80 grids, and each grid corresponds to three preset prior boxes; the divided fusion feature map is slid by using a predictor, and when the predictor slides onto the corresponding grid, the confidence parameters and regression parameters of the prior box are predicted; the CIOU loss function is used to regress and fine-tune the predicted box, and all class information and coordinate information of the predicted box position are decoded; redundant prediction boxes are removed by non-maximum suppression to retain the optimal prediction box, and the final detection result is obtained; other target detection processes are the same as the small target detection process.

5. The method for infrared target detection in complex environment according to claim 1, characterized in that, The expression of the model loss function is: ； ； wherein, denotes the intersection over union of the predicted box and the ground truth box, denotes the distance between the center points of the predicted box and the ground truth box, denotes the Euclidean distance between the center points of the two boxes, denotes the predicted center point coordinates, denotes the ground truth box center point coordinates, denotes the diagonal distance of the minimum rectangular region containing both bounding boxes, is a parameter for balancing the scale, is used to measure the consistency of the aspect ratio, and denote the width and height of the ground truth box, and denote the width and height of the predicted box.

6. The method for infrared target detection in complex environment according to claim 1, characterized in that, The calculation formula of the performance evaluation index is: ； ； wherein, denotes precision, denotes the number of bounding boxes whose predicted value has an intersection over union greater than 0.5 with the true value, denotes the number of bounding boxes whose predicted value has an intersection over union less than or equal to 0.5 with the true value, denotes recall, denotes the number of true values that were not detected.

Images