Infrared remote sensing image target detection method based on YOLOv8

Abstract

The invention discloses an infrared remote sensing image target detection method based on YOLOv8, which belongs to the field of infrared remote sensing image target detection, and specifically comprises that the last but one module and the third module of a backbone network of YOLOv8 are C2fDCN variable convolution modules, that is, the convolution of P3 and P4 detection layers corresponding to the backbone network is replaced by a deformable convolution module, and the C2fDCN variable convolution module is used as a C2fDCN variable convolution module. The attention of the model to small and medium targets is improved; an attention module is added at the check end of the YOLOv8, and the CBAM attention module can adaptively select and adjust the channel weight and the space weight of the feature map, so that important features in the image can be better captured and represented; using a new loss function YIOU to enable the prediction candidate frame to be further regressed to a real object; the infrared aircraft target detection accuracy is enhanced; the invention further discloses electronic equipment and a computer readable storage medium.

Description

[0001] This application is a divisional application with the application date of July 4, 2024, application number: 2024108903132, and application name: A YOLOv8-based infrared remote sensing image target detection method, electronic device, and computer-readable storage medium. Technical Field

[0002] The invention relates to an infrared remote sensing image target detection method based on an improvement of YOLOv8, and belongs to the field of infrared image target detection. Background Art

[0003] Aircraft target detection based on visible light remote sensing images has broad application prospects in the fields of military strikes, aerial target tracking, etc. In recent years, with the development of computers and deep learning methods, target detection methods have achieved great performance improvements.

[0004] However, when the visibility environment changes, it is difficult for traditional visible light remote sensing images to complete the target detection task. Infrared remote sensing images can overcome the limitations of traditional optical imaging systems and provide clear and accurate aircraft images. However, there are currently few studies on the application of deep learning algorithms in the field of infrared remote sensing aircraft images, and under complex conditions, there are certain limitations in the visible light remote sensing target detection method. Infrared remote sensing technology can overcome the limitations of traditional optical imaging systems and provide clear and accurate aircraft images. Summary of the invention

[0005] The purpose of the present invention is to provide an improved infrared remote sensing image aircraft target detection method based on YOLOv8, which has higher accuracy of infrared target detection under complex conditions and better meets the needs of all-weather use.

[0006] In view of the shortcomings of the prior art, the first party of the present application provides an infrared remote sensing image target detection method based on an improved YOLOv8, including obtaining an infrared data set, cleaning the original data to remove fuzzy and repeated data, and finally annotating the obtained image data. The preprocessed infrared image data set is divided into a training set, a validation set, and a test set, wherein the infrared image data set includes infrared images of aircraft under different background environments.

[0007] The penultimate and third C2f modules of the backbone network of YOLOv8 are replaced with C2f_DCN variable convolution modules, that is, the convolutions of the backbone network corresponding to the P3 and P4 detection layers are replaced with deformable convolution modules to improve the model's attention to small and medium-sized targets; an attention module is added to the neck end of YOLOv8. The CBAM attention module can adaptively select and adjust the channel weights and spatial weights of the feature map to better capture and represent important features in the image; the new loss function YIOU is used to enable the predicted candidate box to further regress to the real object.

[0008] The improved method described above is used to obtain a new network model, and the training set is used to iteratively train the new network model to obtain the target weight. The target weight is tested using the test set to obtain the aircraft target detection result.

[0009] The deformable convolution module learns an offset for each point on the convolution kernel, so that the convolution kernel can learn different convolution kernel structures according to different data. After adding the deformable convolution module, because the receptive field of the ship's location has changed adaptively, when regressing the prediction box, the ship detection model can better adjust the regression parameters of the prediction box, so that the model increases its attention to small targets, thereby improving the overall performance of the model.

[0010] The attention module can improve the adaptability of the convolutional neural network to irregularly shaped target objects and expand its receptive field. Embedded in the neck part of YOLOv8, the attention module allows the model to selectively focus on certain parts of the sequence instead of the entire sequence when processing each time step of the sequence. It can adaptively select and adjust the channel weights and spatial weights of the feature map to better capture and represent important features in the image. The location where the CBAM module is added in this paper is after the upsample structure in the PAN-FPN upsampling stage and after each C2f module in the downsampling stage, before the CBS module convolution, that is, before the fusion feature. In the optimized model, the feature attention enhancement operation is performed before the fusion feature, and the model can pay more attention to the information of small targets, enhance the focus on the target, and improve the recognition and positioning accuracy of small targets. In the optimized model, by performing the feature attention enhancement operation before the fusion feature, the model can pay more attention to the information of small targets, enhance the focus on the target, and improve the recognition and positioning accuracy of small targets.

[0011] The new loss function YIOU draws on the idea of ​​wise-IoU and improves the intersection and union operations in the traditional IoU. The specific calculation formula is as follows:

[0012]

[0013] While dynamically adjusting the bounding box regression loss, the penalty for distance, aspect ratio of remote sensing ships and other metrics is weakened. This can better consider multiple factors such as IoU, position, size, shape, etc. between the predicted and true frames of remote sensing ships, thereby improving the accuracy of ship detection. YIoU reduces the second and third terms of the loss function by half, and intervenes in the model's penalty for geometric metrics such as distance and aspect ratio at a lower level, without intervening too much in model training, so that the model has better generalization ability.

[0014] As can be seen from the above, the present application provides an improved infrared aircraft target detection method based on YOLOv8, and improves YOLOv8. In the improved network model, an attention mechanism module is added before the fusion feature. In the optimized model, the feature attention enhancement operation is performed before the fusion feature, and the model can pay more attention to small target information, enhance the focus on the target, and improve the recognition and positioning accuracy of the aircraft target; by adding the deformable convolution module to the backbone network of YOLOv8, the deformable convolution can adaptively adjust the shape and size of the receptive field according to the irregular shape of the target object, thereby improving the performance of the model; by using the new loss function YIoU, YIoU improves the intersection and union operations in the traditional IoU, and while dynamically adjusting the bounding box regression loss, it weakens the penalty for geometric metrics such as distance and aspect ratio. This can better consider multiple factors such as IoU, position, size, shape, etc. between the predicted box and the real box, thereby improving the accuracy of target detection. YIoU halves the second and third terms of the loss function, intervenes in the model's penalty for geometric metrics such as distance and aspect ratio at a lower level, and does not intervene in model training too much, making the model have better generalization capabilities. In short, YIoU has better adaptability and robustness than CIoU and wise-IoU in some specific scenarios, and can better evaluate the performance of ship detection tasks. The predicted candidate box can be further regressed to the real object annotation box, improving the performance of the model; the improved model network is trained, and the trained weights are used for infrared aircraft target detection, and the detection performance is significantly improved, which is suitable for all-weather use scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a flow chart of an infrared remote sensing target detection method based on an improved YOLOv8 provided by the present invention;

[0016] Figure 2 A schematic diagram of a network structure provided by the present invention;

[0017] Figure 3 A schematic diagram of the structure of a C2f_DCN v2 module provided by the present invention;

[0018] Figure 4 A structural schematic diagram of a convolutional attention mechanism CBAM module provided by the present invention. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0020] It will be understood that the term "comprising" when used in this specification and the appended claims indicates that the described features

[0021] The existence of a feature, integer, step, operation, element and / or component does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their collections.

[0022] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0023] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0025] Embodiment 1

[0026] Dataset: The dataset is a large image dataset for target detection in remote sensing images. It can be used to discover and evaluate objects in aerial images. The dataset has pictures, instances, and a total of categories.

[0027] Use Labelmg to annotate the collected images and mark the category and location information of the target.

[0028] The collected images are de-noised and contrast enhanced to improve image quality; the images are enhanced by random cropping, rotation, scaling, etc. to increase the diversity of training samples; the data set is divided into training set, validation set, and test set for model training, tuning, and evaluation. In order to test the excellent performance and generalization ability of infrared images in all-weather use, the training set, validation set, and test set are composed of infrared images with different lighting conditions at different times.

[0029] In this embodiment, we use accuracy, recall, mean average precision (mAP) and mean average precision as evaluation indicators. Among them, accuracy refers to the ratio of the number of samples correctly predicted by the model as positive examples to the number of all samples predicted as positive examples, reflecting the accuracy of the model in positive example prediction; recall refers to the ratio of the number of samples correctly identified as positive examples by the model to the number of samples that are actually positive examples, representing the recall rate of the model for positive examples. mAP is an indicator for evaluating the performance of the model in each category, which is obtained by calculating the mean precision (AP) of each category and averaging the AP of all categories. The calculation process includes sorting the prediction results of each category, calculating the area under the precision-recall curve (ie AP), and finally averaging the AP of all categories to obtain mAP.

[0030] In this example, a lightweight version of YOLOv8, YOLOv8s, is selected as the baseline model to improve its model structure in order to enhance the detection capability of small targets.

[0031] The research content of the present invention includes improving the receptive field of the detection point on the feature map and adding a deformable convolution module to the backbone network. First, the input feature map is projected, segmented and connected in the depth dimension to obtain a set of rich intermediate feature sets. For the path using convolution, the intermediate feature set first expands the channel through the full connection of the channel layer, then shifts it, and then aggregates it into the corresponding dimension. This path can well extract local feature information in remote sensing images and fuse features at different levels. The path can eventually obtain H*W*C feature output. For the path using self-attention, the intermediate features are aggregated into N groups, each group contains three feature maps and each group is a feature generated by different 1*1 convolutions, and then these three feature maps are input into the multi-head self-attention module as query key values. This path not only considers the overall situation of the image but also pays full attention to the area where the target exists. The path can eventually obtain H*W*C feature output. Finally, the final output result is obtained through the concat operation.

[0032] The attention mechanism added at the neck end aggregates spatial information using the maximum pooling operation and the average pooling operation for the input feature map, and outputs two C-dimensional pooled feature maps, F avg and Fmax (2) F avg and F max Input into the multilayer perceptron MLP containing the hidden layer, and obtain two channel attention maps with parameters of 1×1×C. In order to reduce the number of parameters, the hidden layer neurons need to compress the input information. The number of neurons is C / r, where r is called the compression ratio. (3) Add the output of the previous step to obtain the channel attention map M c . M c It can be expressed by the following formula:

[0033]

[0034] The IoU criterion is designed to enable the network to adaptively adjust the proportion of each part of the loss function at different stages. Compared with CIoU and Wise IoU, the YIoU defined in this paper can first dynamically adjust the bounding box regression loss like Wise-IoU. When the model is in the early stage of training, the IoU between the predicted candidate box and the real object annotation box is relatively small. At this time, the model should focus on improving the IoU between the predicted candidate box and the real object annotation box. The RWIoU in the first term RWIoU×LIoU of the above IoU loss can effectively amplify the model's small penalty for IoU. When it comes to the late stage of training, the IoU between the predicted candidate box and the real object annotation box basically maintains a high degree of overlap and basically does not change much. At this time, the LIoU in the first term RWIoU×LIoU of the above IoU loss decays to a smaller value. At this time, the model will automatically focus on the regression of the candidate box on the center point and the regression of the aspect ratio, so that the predicted candidate box can be further regressed to the real object annotation box, improving the performance of the model.

[0035] The experimental environment of this experiment is: processor Intel Core i9-12900H; memory is 64GB DDR4 (3200MHZ); graphics card GeForce RTX 3090; storage hard disk is 1024GB PCIe 4.0NVMe solid-state drive; operating system is Ubuntu 20.04 development environment: Python 3.9.17pytorch 2.0.1torchvision 0.15.2.

[0036] In summary, this paper proposes an infrared target detection method based on YOLOv8. Previous studies mainly focused on horizontal aircraft detection from remote sensing image sequences, which resulted in uneven target distribution, complex background shapes, and difficult calculation of target classification and positioning losses, leading to poor target detection performance. In order to solve this problem, we proposed a new infrared remote sensing image detection model based on YOLOv8. This invention provides an innovative method for efficient and accurate detection of infrared remote sensing, has important practical application value, and can be applied to real-time target detection tasks.

[0037] The above embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for infrared remote sensing image target detection based on YOLOv8, characterized in that: Here are the steps: S1: Get the dataset; S2: Use Labelmg to annotate the collected images and mark the category and location information of the target. S3: Perform denoising and contrast enhancement on the collected images to improve image quality; enhance the images by random cropping, rotation, scaling, etc. to increase the diversity of training samples; S4: Divide the dataset into training set, validation set and test set for model training, tuning and evaluation; In order to test the excellent performance and generalization ability of infrared images in all-weather use, the training set, validation set, and test set are composed of infrared images with different lighting conditions at different times; S4: For the path using self-attention, the intermediate features are aggregated into N groups, each group contains three feature maps and each group is a feature generated by a different 1*1 convolution, and then these three feature maps are input into the multi-head self-attention module as query key values; S5: The input feature map is projected, segmented and connected in the depth dimension to obtain a set of rich intermediate feature sets. For the path using convolution, the intermediate feature set is first expanded through the full connection of the channel layer, then displaced, and then aggregated into the corresponding dimension; for the path using self-attention, the intermediate features are aggregated into N groups, each group contains three feature maps and each group is a feature generated by a different 1*1 convolution, and then these three feature maps are input into the multi-head self-attention module as query key values; finally, the final output result is obtained through the concat operation.

2. The infrared remote sensing image target detection method based on YOLOv8 according to claim 1, characterized in that: Accuracy, recall, mean average precision (mAP) and average precision were used as evaluation indicators.

3. The infrared remote sensing image target detection method based on YOLOv8 according to claim 2, characterized in that: Precision refers to the ratio of the number of samples correctly predicted as positive examples by the model to the number of all samples predicted as positive examples; recall refers to the ratio of the number of samples correctly identified as positive examples by the model to the number of samples that are actually positive examples; average precision is an indicator for evaluating the performance of the model in each category, which is obtained by calculating the average precision (AP) of each category and averaging the AP of all categories.

Images