Infrared image multi-layer feature enhancement detection integrated method, device and equipment

Abstract

This application relates to an integrated method, apparatus, and device for multi-layer feature enhancement and detection of infrared images. The method includes: acquiring low-quality infrared image samples; inputting the low-quality infrared image samples into an integrated enhancement and detection model; the integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network; the improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network; training the integrated enhancement and detection model based on a pre-constructed loss function and the low-quality infrared image samples to obtain a trained integrated enhancement and detection model; and performing infrared image target detection using the trained integrated enhancement and detection model. This method can significantly improve the accuracy and efficiency of weak target detection under complex weather conditions on edge platforms, meeting the needs of airborne applications.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an integrated method, apparatus and device for multi-layer feature enhancement and detection of infrared images. Background Technology

[0002] Infrared detection technology, due to its excellent passive detection capabilities and all-weather performance, has become an important component of modern reconnaissance and surveillance systems. This technology can detect targets without emitting any radiation source, thus significantly improving the concealment of the detected subject. Especially in airborne applications, infrared detection is less affected by lighting conditions, allowing for continuous surveillance and target tracking missions at night or in adverse weather conditions. However, airborne applications typically use uncooled infrared sensors. Due to limitations in imaging devices and design, imaging in complex weather conditions suffers from insufficient brightness, loss of detail, and reduced contrast, affecting subsequent infrared target detection and identification performance.

[0003] In addition, traditional cloud processing models suffer from high latency, while real-time on-site processing typically requires significant computing resources, leading to increased equipment size and weight, which in turn limits the flexibility and battery life of airborne infrared detection platforms.

[0004] Compared to traditional cloud processing, edge computing can significantly reduce data transmission latency and provide real-time information processing capabilities. This advantage is particularly critical for airborne platforms, which require systems to respond quickly and process large amounts of sensor data to support emergency decision-making and immediate task adjustments. Therefore, how to significantly improve infrared imaging performance, accuracy, and efficiency of weak target detection in complex weather conditions on edge platforms to meet the application requirements of airborne infrared target detection and identification has become an important problem that urgently needs to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide an integrated method, apparatus, and device for infrared image multi-layer feature enhancement detection to address the aforementioned technical problems.

[0006] A method for integrated detection of multi-layer feature enhancement in infrared images, the method comprising:

[0007] Low-quality infrared image samples are acquired and input into an integrated enhancement and detection model. The integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network. The YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network.

[0008] The low-quality infrared image sample is enhanced by the shallow feature enhancement network to obtain a high-quality infrared image. The high-quality infrared image is then enhanced by the deep feature enhancement network to obtain an enhanced feature map. The enhanced feature map is then convolved by the first convolutional layer to obtain a first feature map.

[0009] The backbone network extracts multiple features from the high-quality infrared image to obtain a second feature map. The second feature map is then convolved by the second convolutional layer. The first feature map and the convolved second feature map are then fused by the stitching layer to obtain a fused feature map. The fused feature map is then predicted by the head network to obtain a prediction result. The prediction result is then displayed on the high-quality infrared image to output the target detection result.

[0010] The enhanced detection integrated model is trained based on the pre-constructed loss function and the low-quality infrared image samples to obtain the trained enhanced detection integrated model.

[0011] Infrared image target detection is performed using a trained integrated augmentation and detection model.

[0012] In one embodiment, the shallow feature enhancement network includes a generator network and a discriminator network. The discriminator network assists in the training of the generator network, using the pre-trained generator network to enhance the features of the low-quality infrared image samples. The discriminator network includes multiple sequentially connected coding layers, fully connected layers, and an output layer. The generator network includes a first shallow feature extraction module and a first deep feature extraction module, both sequentially connected. Each module includes at least two coding layers. The shallow features output from the first coding layer in the first shallow feature extraction module are concatenated with the output features from the second-to-last coding layer in the first deep feature extraction module to obtain concatenated features. These concatenated features are then weighted and fused with the shallow features output from the second-to-last coding layer in the first shallow feature extraction module to obtain weighted fused features. These weighted fused features are then input into the last coding layer of the first deep feature extraction module to output a high-quality infrared image.

[0013] In one embodiment, the step of obtaining the pre-trained shallow feature enhancement network includes: acquiring high-quality infrared image samples corresponding to the low-quality infrared image samples; randomly inputting the high-quality infrared image output by the generator network and the high-quality infrared image samples into the discriminator network for discrimination, and outputting the corresponding discrimination result; training the generator network according to the total image enhancement loss function, the low-quality infrared image samples, the high-quality infrared image samples, and the discrimination result output by the discriminator network to obtain the pre-trained shallow feature enhancement network; the total image enhancement loss function includes pixel difference loss, adversarial loss, and total variational loss.

[0014] In one embodiment, the deep feature enhancement network includes a second shallow feature extraction module and a second deep feature extraction module connected in sequence. The second deep feature extraction module includes at least three coding layers, wherein the enhanced feature map is obtained based on the deep features output by the penultimate coding layer, the penultimate coding layer and the last coding layer in the second deep feature extraction module.

[0015] In one embodiment, the step of obtaining the pre-trained deep feature enhancement network includes: concatenating the deep feature network with the pre-trained head network; freezing the weights of the head network; training the deep feature network according to the total loss function for object detection to obtain the pre-trained deep feature enhancement network; the total loss function for object detection includes bounding box regression loss, object confidence loss, and category loss.

[0016] In one embodiment, the total loss function for target detection is:

[0017] L dt =λ5L obj +λ6L cls +λ7L CIoU

[0018] Among them, L dt Let L be the total loss function for object detection, where λ5, λ6, and λ7 are the weights of the object confidence loss, class loss, and bounding box regression loss, respectively. obj For target confidence loss, N is the total number of targets in the sample, t i c represents the IoU value between the ground truth bounding box and the predicted bounding box. i To predict confidence levels, L cls For category loss, y i For the true category label, p i L is the category prediction value. CIoU For bounding box regression loss, IoU is the crossover ratio, ρ 2 (b,bgt Let d be the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and let d be the diagonal distance of the smallest rectangular region that can simultaneously contain both the predicted and ground truth bounding boxes. w and h are the width and height of the predicted bounding box, respectively, and w and h are the width and height of the ground truth bounding box, respectively.

[0019] In one embodiment, the backbone network includes a first feature extraction module and three sets of second feature extraction modules connected in sequence. The first feature extraction module includes two encoding layers and an L-ELAN module connected in sequence. The second feature extraction module includes an L-ELAN module and an MP module. The L-ELAN module includes a multi-scale convolution module, a multi-scale feature fusion module, and a fused feature extraction module. The multi-scale convolution module includes a first-scale convolution sub-module and a second-scale convolution sub-module. The first-scale convolution sub-module includes at least one encoding layer. The second-scale convolution sub-module includes multiple encoding layers with the same convolution kernel. The output features of each encoding layer in the multi-scale convolution module are fused by the multi-scale feature fusion module and then used as the input of the fused feature extraction module. The fused feature extraction module extracts features to obtain the output features of the second feature extraction module.

[0020] In one embodiment, the augmentation detection ensemble model is trained based on a pre-constructed loss function and low-quality infrared image samples to obtain a trained augmentation detection ensemble model. This includes: freezing only the pre-trained weights of the shallow feature enhancement network and the deep feature enhancement network, and training the augmentation detection ensemble model using the total target detection loss function and low-quality infrared image samples; freezing only the weights of the improved YOLOv7 target detection network, and training the augmentation detection ensemble model using the total target detection loss function and low-quality infrared image samples to obtain a trained augmentation detection ensemble model.

[0021] An integrated device for infrared image multi-layer feature enhancement and detection, the device comprising:

[0022] The sample input module is used to acquire low-quality infrared image samples and input the low-quality infrared image samples into the integrated enhancement and detection model; the integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network; the improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network.

[0023] The feature enhancement module is used to enhance the features of the low-quality infrared image sample through the shallow feature enhancement network to obtain a high-quality infrared image, enhance the features of the high-quality infrared image through the deep feature enhancement network to obtain an enhanced feature map, and perform convolution processing on the enhanced feature map through the first convolutional layer to obtain a first feature map.

[0024] The enhanced detection integrated module is used to extract multiple features from the high-quality infrared image through the backbone network to obtain a second feature map, perform convolution processing on the second feature map through the second convolutional layer, fuse the first feature map and the convolutional second feature map through the stitching layer to obtain a fused feature map, predict the fused feature map through the head network to obtain a prediction result, display the prediction result on the high-quality infrared image, and output the target detection result.

[0025] The model training module is used to train the integrated enhancement detection model based on a pre-constructed loss function and the low-quality infrared image samples to obtain a trained integrated enhancement detection model.

[0026] The target detection module is used to perform target detection in infrared images using a trained integrated augmented detection model.

[0027] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0028] Low-quality infrared image samples are acquired and input into an integrated enhancement and detection model. The integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network. The improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network.

[0029] The low-quality infrared image sample is enhanced by the shallow feature enhancement network to obtain a high-quality infrared image. The high-quality infrared image is then enhanced by the deep feature enhancement network to obtain an enhanced feature map. The enhanced feature map is then convolved by the first convolutional layer to obtain a first feature map.

[0030] The backbone network extracts multiple features from the high-quality infrared image to obtain a second feature map. The second feature map is then convolved by the second convolutional layer. The first feature map and the convolved second feature map are then fused by the stitching layer to obtain a fused feature map. The fused feature map is then predicted by the head network to obtain a prediction result. The prediction result is then displayed on the high-quality infrared image to output the target detection result.

[0031] The enhanced detection integrated model is trained based on the pre-constructed loss function and the low-quality infrared image samples to obtain the trained enhanced detection integrated model.

[0032] Infrared image target detection is performed using a trained integrated augmentation and detection model.

[0033] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0034] Low-quality infrared image samples are acquired and input into an integrated enhancement and detection model. The integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network. The improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network.

[0035] The low-quality infrared image sample is enhanced by the shallow feature enhancement network to obtain a high-quality infrared image. The high-quality infrared image is then enhanced by the deep feature enhancement network to obtain an enhanced feature map. The enhanced feature map is then convolved by the first convolutional layer to obtain a first feature map.

[0036] The backbone network extracts multiple features from the high-quality infrared image to obtain a second feature map. The second feature map is then convolved by the second convolutional layer. The first feature map and the convolved second feature map are then fused by the stitching layer to obtain a fused feature map. The fused feature map is then predicted by the head network to obtain a prediction result. The prediction result is then displayed on the high-quality infrared image to output the target detection result.

[0037] The enhanced detection integrated model is trained based on the pre-constructed loss function and the low-quality infrared image samples to obtain the trained enhanced detection integrated model.

[0038] Infrared image target detection is performed using a trained integrated augmentation and detection model.

[0039] The aforementioned integrated method, apparatus, and equipment for infrared image multi-layer feature enhancement and detection perform image enhancement and detection on low-quality infrared image samples using an integrated enhancement and detection model. Specifically, a shallow feature enhancement network extracts shallow features from the low-quality infrared image to obtain a high-quality infrared image. This high-quality infrared image is then input into a deep feature enhancement network and a backbone network, respectively. The output feature maps of the two networks are then fused, and the fused features are input into a head network for weak target detection. This method can improve the quality of infrared images and the accuracy of weak target detection, while also increasing the detection processing speed on edge-end infrared devices. Attached Figure Description

[0040] Figure 1 This is an application scenario diagram of an integrated infrared image multi-layer feature enhancement and detection method in one embodiment;

[0041] Figure 2 This is a schematic diagram of the enhanced detection integrated model in one embodiment;

[0042] Figure 3 This is a schematic diagram of the structure of a shallow feature enhancement network in one embodiment, wherein Figure (a) is the generator network structure and Figure (b) is the discriminator network structure;

[0043] Figure 4 This is a schematic diagram of the structure of a deep feature enhancement network in one embodiment;

[0044] Figure 5 Figure (a) is a schematic diagram of the structure of a multi-feature enhanced YOLOv7 target detection network in one embodiment. Figure (b) is a schematic diagram of the backbone network structure, and Figure (c) is a schematic diagram of the L-ELAN module structure.

[0045] Figure 6 This is a structural block diagram of an integrated infrared image multi-layer feature enhancement and detection device in one embodiment;

[0046] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] In one embodiment, such as Figure 1 As shown, an integrated method for multi-layer feature enhancement and detection of infrared images is provided, including the following steps:

[0049] Step 102: Obtain low-quality infrared image samples and input them into the integrated enhancement detection model.

[0050] The integrated augmented detection model includes a pre-trained low-level feature maps enhancement network, a pre-trained high-level feature maps enhancement network, a first convolutional layer, and an improved YOLOv7 object detection network. The improved YOLOv7 object detection network includes a backbone network (Detect net Backbone), a second convolutional layer, a concatenation layer, and a head network (Detect net Head).

[0051] This paper applies to low-cost edge infrared devices. Edge devices, located at the edge of an information processing system, are primarily responsible for data collection, processing, and transmission, as well as communication with other devices. However, in complex weather conditions, infrared imaging is poor, resulting in low-quality infrared images containing targets collected by edge infrared devices. Furthermore, edge infrared devices have limited computing resources. These two factors negatively impact the accuracy and efficiency of infrared detection of weak targets. Therefore, this paper designs an integrated augmentation and detection model that combines feature enhancement for low-quality infrared images with high-precision detection of weak targets.

[0052] like Figure 2 The schematic diagram of the integrated augmented detection model shows that a low-quality infrared image is input, processed by a shallow feature enhancement network, and output as a high-quality infrared image. Then, the high-quality infrared image is simultaneously input into both the target detection network backbone and the deep feature enhancement network for feature extraction and enhancement. Based on this, the features extracted by the target detection network backbone and the enhanced features output by the deep feature enhancement network are segmented and fused along the channel direction. Finally, the fused features are input into the head part of the target detection network for prediction, and the prediction result is displayed on the enhanced high-quality image.

[0053] Considering the limited computing resources and power supply of edge devices, a lightweight design was implemented for the improved YOLOv7 target detection network in the detection section. In the backbone part, multiple features were extracted, enhanced, and fused from high-quality infrared images, resulting in images with rich semantic information input to the head. Therefore, the performance of the lightweight network did not significantly decrease compared to designs with deeper convolutional layers.

[0054] The integrated enhancement and detection model solves the problem of separating image enhancement and object detection tasks, while taking into account both feature enhancement and visual effect enhancement. It improves the performance of the object detection network while making it easier for operators to visualize.

[0055] Step 104: Perform feature enhancement on low-quality infrared image samples through a shallow feature enhancement network to obtain a high-quality infrared image; perform feature enhancement on the high-quality infrared image through a deep feature enhancement network to obtain an enhanced feature map; and perform convolution processing on the enhanced feature map through a first convolutional layer to obtain a first feature map.

[0056] Shallow feature enhancement networks and deep feature enhancement networks, used for image enhancement, focus on extracting shallow and deep features, respectively. The shallow feature enhancement network is a generative adversarial network (GAN). GANs can significantly improve infrared image enhancement by training the discriminator and generator networks in an adversarial manner. However, the network has a large number of training parameters, consumes significant resources at the edge deployment stages, and has high computational complexity. To reduce the number of training parameters, this paper designs a shallow feature enhancement network by reducing the structural complexity of the generator network, thereby improving feature extraction efficiency and reducing the number of layers required in the network.

[0057] It's important to note that image augmentation and object detection tasks are typically separated, which prevents image augmentation networks from effectively optimizing parameters to improve detection performance. To address this issue, this paper designs a deep feature augmentation network (DNF) to enhance the features of interest to the object detection network. During training, the DNF is concatenated with the head of a pre-trained object detection network, using a frozen weighted head as an aid to optimize its weights. This ensures that the image features output by the DNF during inference are beneficial to the object detection network, organically combining the image augmentation and object detection tasks. During inference, the auxiliary training head is removed, and the DNF and shallow feature augmentation networks are fed in parallel into the object detection network head for prediction.

[0058] Step 106: Extract multiple features from the high-quality infrared image through the backbone network to obtain the second feature map. Perform convolution processing on the second feature map through the second convolutional layer. Perform feature fusion on the first feature map and the convolutional second feature map through the stitching layer to obtain the fused feature map. Predict the fused feature map through the head network to obtain the prediction result. Display the prediction result on the high-quality infrared image and output the target detection result.

[0059] In the target detection stage, YOLOv7 is used as the basic framework. Considering the limited computing resources and power supply of edge devices, a lightweight design is implemented in the network backbone.

[0060] Step 108: Train the integrated augmentation and detection model based on the pre-constructed loss function and low-quality infrared image samples to obtain the trained integrated augmentation and detection model.

[0061] The shallow and deep feature enhancement networks based on generative adversarial approaches are pre-trained, and then jointly trained with the multi-feature enhancement YOLOv7 object detection network to obtain the final model.

[0062] Step 110: Perform infrared image target detection using the trained integrated augmentation and detection model.

[0063] The aforementioned integrated method for multi-layer feature enhancement and detection of infrared images utilizes an integrated enhancement and detection model to perform image enhancement and detection on low-quality infrared image samples. Specifically, a shallow feature enhancement network extracts shallow features from the low-quality infrared image to obtain a high-quality infrared image. This high-quality infrared image is then input into both a deep feature enhancement network and a backbone network. The output feature maps of both networks are then fused, and the fused features are input into a head network for weak target detection. This method improves the quality of infrared images and the accuracy of weak target detection, while also increasing the detection processing speed on edge infrared devices.

[0064] In one embodiment, such as Figure 3 As shown, a schematic diagram of a shallow feature enhancement network is provided, wherein, Figure 3 (a) shows the generator network structure. Figure 3 (b) is the discriminator network structure. The shallow feature enhancement network includes a generator network and a discriminator network. The discriminator network is used to assist the generator network in training and to enhance the features of low-quality infrared image samples using the pre-trained generator network. The discriminator network includes multiple coding layers, fully connected layers and an output layer connected in sequence. The generator network includes a first shallow feature extraction module and a first deep feature extraction module connected in sequence. The first shallow feature extraction module and the first deep feature extraction module include at least two coding layers. The shallow features output by the first coding layer in the first shallow feature extraction module are concatenated with the output features of the second-to-last coding layer in the first deep feature extraction module to obtain concatenated features. The concatenated features are then weighted and fused with the shallow features output by the second coding layer in the first shallow feature extraction module to obtain weighted fused features. The weighted fused features are then input into the last coding layer of the first deep feature extraction module to output a high-quality infrared image.

[0065] In this embodiment, as Figure 3 The generator network shown in (a) concatenates the output features of the first convolutional layer with the output of the fifth layer, preserving the shallow features of the image, and then weights and fuses them with the output features of the second convolutional layer. Compared with a simple stacked convolutional layer structure, this can extract multi-level image features without introducing high computational complexity, improving the efficiency of convolutional layer feature extraction. Compared with the simple skip connection weighted addition method, the generator network in this paper reduces the loss of shallow image features. The generator network contains a total of 6 convolutional layers, extracting image features at different scales, which facilitates subsequent difference comparison with high-quality images in the discriminator network. Figure 3 (b) shows the discriminator network, which employs four convolutional layers and one fully connected neural network, ultimately outputting the discrimination result through a softmax layer. Enhanced images and high-quality infrared image samples generated by the generator are randomly input into the discriminator network, resulting in either an enhanced image (fake) or a high-quality image (real). The discriminator network only assists the generator network during the training phase; during the inference phase, only the generator network is used, and the discriminator network does not introduce additional computational complexity.

[0066] In one embodiment, the step of obtaining a pre-trained shallow feature enhancement network includes: acquiring high-quality infrared image samples corresponding to low-quality infrared image samples; randomly inputting the high-quality infrared image output by the generator network and the high-quality infrared image samples into a discriminator network for discrimination, and outputting the corresponding discrimination result; training the generator network based on the total image enhancement loss function, the low-quality infrared image samples, the high-quality infrared image samples, and the discrimination result output by the discriminator network to obtain a pre-trained shallow feature enhancement network; the total image enhancement loss function includes pixel difference loss, adversarial loss, and total variational loss.

[0067] In this embodiment, to comprehensively consider the differences in local and overall features between low-quality and high-quality images, a multi-scale loss function is designed, including pixel difference loss, adversarial loss, similarity loss, and total variational loss, in order to improve the performance of the image enhancement network.

[0068] (1) Pixel difference loss

[0069] Existing infrared image enhancement networks mostly use L2 loss to measure local feature differences in images. This loss does not consider the relationships between pixels, resulting in inaccurate measurement of differences between images. Therefore, this paper designs a loss function based on inter-pixel region relationships, incorporating the relationships between pixels into the loss function to improve the accuracy of image difference measurement. This paper uses a Gaussian mixture model (GMM)-based method to determine the relationships between pixels. For both the enhanced image and the target high-quality image, a GMM is used to fit the entire image. When calculating the pixel difference loss, the weights are obtained through the similarity of the GMM between pixels in the two images.

[0070]

[0071] Where p(i,j) is a Gaussian mixture model fitted based on the image, and π k The mixing coefficients are calculated by fitting specific images. N(i,j) is a two-dimensional Gaussian distribution. The KL divergence characterizes the similarity between two probability distributions; the higher the similarity, the smaller the KL divergence value. The KL divergence of the Gaussian mixture model (pk1,pk2) using the k-th pair of enhanced and target high-quality images is defined as the weight ωk of the Gaussian mixture model, and its calculation method is as follows:

[0072]

[0073] The pixel difference loss is obtained as follows:

[0074]

[0075] Where C, H, and W represent the number, height, and width of the image, and G(I)(i,j) and T(i,j) represent the pixel values ​​of the enhanced image and the target high-quality image, respectively.

[0076] (2) Combating losses

[0077] The discriminator extracts and identifies texture information, performs binary classification on randomly input enhanced images and target high-quality images, determines whether an image is genuine, and outputs the estimated result. In the initial training stage, considering the poor performance of the generator, this paper uses maximizing logD(G(I)) instead of minimizing logD(1-G(I)) to increase the update gradient of the generator parameters, thereby accelerating network training. The final adversarial loss function is:

[0078]

[0079] Where G and D represent the generator and discriminator networks, respectively, I represents the low-quality infrared image, G(I) represents the augmented image (i.e., the high-quality infrared image), T represents the target high-quality image, and i represents the total number of training images. The discriminator is pre-trained on image pairs {augmented image, target high-quality image}, and then jointly trained with the generator network to minimize the adversarial loss.

[0080] (3) Total variational loss

[0081] To meet the requirements of infrared image display, this paper introduces a total variational loss into the loss function. The aim is to eliminate noise introduced during image enhancement and smooth the image to satisfy the visual perception needs of the human eye. The total variational loss is:

[0082]

[0083] Where C, H, and W are the dimensions of the enhanced image G(I).

[0084] The overall image enhancement loss function designed in this paper is a weighted sum of the above loss functions:

[0085] L eh =λ1L pd +λ2L ad +λ3L tv

[0086] λ1, λ2, and λ3 are weighting parameters, whose optimal values ​​are determined experimentally.

[0087] In one embodiment, such as Figure 4 As shown, a schematic diagram of a deep feature enhancement network is provided. The deep feature enhancement network includes a second shallow feature extraction module and a second deep feature extraction module connected in sequence. The second deep feature extraction module includes at least three encoding layers. The enhanced feature map is obtained based on the deep features output from the penultimate, second-to-last, and last encoding layers in the second deep feature extraction module. In this embodiment, as... Figure 4 As shown, the deep feature enhancement network contains five convolutional layers, with convolutional kernels of different depths designed to extract image features at different scales. Then, the output features of layers 3, 4, and 5 are used as input to the head part of the object detection network.

[0088] In one embodiment, the steps of obtaining a pre-trained deep feature enhancement network include: concatenating the deep feature network with a pre-trained head network; freezing the weights of the head network; training the deep feature network according to the total loss function for object detection; and obtaining the pre-trained deep feature enhancement network. The total loss function for object detection includes bounding box regression loss, object confidence loss, and class loss.

[0089] In one embodiment, the total loss function for object detection is:

[0090] L dt =λ5L obj +λ6L cls +λ7L CIoU

[0091] Among them, L dt Let L be the total loss function for object detection, where λ5, λ6, and λ7 are the weights of the object confidence loss, class loss, and bounding box regression loss, respectively. obj For target confidence loss, N is the total number of targets in the sample, t i c represents the IoU value between the ground truth bounding box and the predicted bounding box. i To predict confidence levels, L cls For category loss, y i For the true category label, p i L is the category prediction value. CIoU For bounding box regression loss, IoU is the crossover ratio, ρ 2 (b,b gt Let d be the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and let d be the diagonal distance of the smallest rectangular region that can simultaneously contain both the predicted and ground truth bounding boxes. w and h are the width and height of the predicted bounding box, respectively, and w and h are the width and height of the ground truth bounding box, respectively.

[0092] In this embodiment, the loss function of the YOLO series is used in this part, and only the loss of positive samples is considered. The loss function includes the following three parts: CIoU loss (boundary box regression loss), Objectness loss (object confidence loss), and Classification loss (classification loss). The bounding box regression loss considers three geometric parameters: overlap area, center point distance, and aspect ratio. The object confidence loss is defined as the degree of overlap between the ground truth bounding box and the predicted bounding box of a positive sample, and is calculated using BCE loss. The classification loss is defined as the similarity between the ground truth class and the predicted class of a positive sample, and is calculated using BCE loss. The total object detection loss is defined as the weighted sum of the above losses. The initial weight values ​​refer to the settings of the YOLO series, and are subsequently adjusted through experiments to determine the optimal values.

[0093] In one embodiment, such as Figure 5 As shown, a schematic diagram of a multi-feature enhanced YOLOv7 object detection network is provided, wherein... Figure 5 (a) is a schematic diagram of the backbone network structure. Figure 5 (b) is a schematic diagram of the L-ELAN module structure. Figure 5(c) is a schematic diagram of the MP module structure. The backbone network includes a first feature extraction module and three sets of second feature extraction modules connected in sequence. The first feature extraction module includes two coding layers and an L-ELAN module connected in sequence. The second feature extraction module includes an L-ELAN module and an MP module. The L-ELAN module includes a multi-scale convolution module, a multi-scale feature fusion module, and a fusion feature extraction module. The multi-scale convolution module includes a first-scale convolution sub-module and a second-scale convolution sub-module. The first-scale convolution sub-module includes at least one coding layer. The second-scale convolution sub-module includes multiple coding layers with the same convolution kernel. The output features of each coding layer in the multi-scale convolution module are fused by the multi-scale feature fusion module and then used as the input of the fusion feature extraction module. The fusion feature extraction module extracts features to obtain the output features of the second feature extraction module.

[0094] In this embodiment, we removed two convolutional layers from the backbone front end to reduce computational cost, making it more suitable for edge device deployment and inference. Secondly, we replaced the ELAN structure with an L-ELAN structure to further reduce the number of model parameters without significantly reducing network performance. ELAN (Efficient Layer Aggregation Networks) is an efficient network design architecture that enables deep networks to learn and converge effectively by controlling the shortest and longest gradient paths. Based on the ELAN structure, this paper proposes an L-ELAN (Light-ELAN) structure. The L-ELAN structure maintains the original ELAN design architecture while achieving lightweight network design and high-efficiency feature extraction by removing redundant convolutional layers and expanding the convolutional kernel scale, without significantly reducing the network's learning ability. Specifically, the L-ELAN structure removes two redundant convolutional layers from the group convolutions to reduce the computational cost of the convolutional groups, adds an additional 4×4 convolutional layer to extract convolutions at different scales, and finally concatenates the feature maps calculated from each computational block. In this way, the L-ELAN structure can not only maintain the original ELAN design architecture, but also guide different groups of computational blocks to learn more diverse features while reducing model complexity, without significantly reducing the overall performance of the network.

[0095] In one embodiment, training the augmentation detection fusion model based on a pre-constructed loss function and low-quality infrared image samples to obtain the trained augmentation detection fusion model includes: first, freezing only the pre-trained weights of the shallow feature enhancement network and the deep feature enhancement network, and training the augmentation detection fusion model using the total target detection loss function and low-quality infrared image samples; then, freezing only the weights of the improved YOLOv7 target detection network, and training the augmentation detection fusion model using the total target detection loss function and low-quality infrared image samples to obtain the trained augmentation detection fusion model.

[0096] In this embodiment, the pre-trained network includes a shallow feature enhancement network and a deep feature enhancement network. The shallow feature enhancement network is pre-trained, and the network weights are optimized through backpropagation using the overall image enhancement loss function. The pre-trained weights are then saved to obtain the pre-trained shallow feature enhancement network model. To address the problem that the image enhancement network cannot effectively optimize parameters towards improving detection performance due to the separation of image enhancement and object detection tasks, as follows... Figure 4 As shown, the head part of the deep feature enhancement network and the object detection network are cascaded. Backpropagation is performed using the total loss function for object detection to optimize the network weights. Finally, the pre-trained weights of the deep feature enhancement network are saved, resulting in a pre-trained deep feature enhancement network model. Next, a joint training of the infrared image multi-layer feature enhancement and detection integrated network is performed, where the shallow and deep feature enhancement networks use pre-trained weights. The weights of the deep feature enhancement network are frozen to ensure the visual enhancement effect of the shallow feature enhancement network. Then, backpropagation is performed using the total loss function for object detection to optimize the weight parameters of the entire network. Finally, the trained weights are saved, resulting in an infrared image multi-layer feature enhancement and detection integrated network model for edge applications. The proposed method significantly improves the quality of infrared images and the accuracy of weak target detection, while achieving a detection processing speed of over 15fps on edge devices, meeting the requirements of airborne applications.

[0097] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0098] In one embodiment, such as Figure 6 As shown, an integrated device for multi-layer feature enhancement and detection of infrared images is provided, comprising:

[0099] The sample input module 602 is used to acquire low-quality infrared image samples and input the low-quality infrared image samples into the integrated enhancement and detection model. The integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network. The improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network.

[0100] The feature enhancement module 604 is used to enhance the features of low-quality infrared image samples through a shallow feature enhancement network to obtain a high-quality infrared image, enhance the features of the high-quality infrared image through a deep feature enhancement network to obtain an enhanced feature map, and perform convolution processing on the enhanced feature map through a first convolutional layer to obtain a first feature map.

[0101] The enhanced detection integrated module 606 is used to extract multiple features from a high-quality infrared image through a backbone network to obtain a second feature map, perform convolution processing on the second feature map through a second convolutional layer, fuse the first feature map and the convolutional second feature map through a stitching layer to obtain a fused feature map, predict the fused feature map through a head network to obtain a prediction result, display the prediction result on the high-quality infrared image, and output the target detection result.

[0102] The model training module 608 is used to train the integrated augmentation and detection model based on a pre-constructed loss function and low-quality infrared image samples to obtain a trained integrated augmentation and detection model.

[0103] The target detection module 610 is used for infrared image target detection using a trained integrated augmented detection model.

[0104] In one embodiment, the shallow feature enhancement network includes a generator network and a discriminator network. The discriminator network assists in the training of the generator network, using the pre-trained generator network to enhance the features of low-quality infrared image samples. The discriminator network includes multiple sequentially connected coding layers, fully connected layers, and an output layer. The generator network includes a first shallow feature extraction module and a first deep feature extraction module, both sequentially connected. Each module includes at least two coding layers. The shallow features output from the first coding layer in the first shallow feature extraction module are concatenated with the output features from the second-to-last coding layer in the first deep feature extraction module to obtain concatenated features. These concatenated features are then weighted and fused with the shallow features output from the second-to-last coding layer in the first shallow feature extraction module to obtain weighted fused features. These weighted fused features are then input into the last coding layer of the first deep feature extraction module to output a high-quality infrared image.

[0105] In one embodiment, the step of obtaining the pre-trained shallow feature enhancement network includes: acquiring high-quality infrared image samples corresponding to low-quality infrared image samples; randomly inputting the high-quality infrared image output by the generator network and the high-quality infrared image samples into the discriminator network for discrimination, and outputting the corresponding discrimination result; training the generator network according to the total image enhancement loss function, the low-quality infrared image samples, the high-quality infrared image samples, and the discrimination result output by the discriminator network to obtain the pre-trained shallow feature enhancement network; the total image enhancement loss function includes pixel difference loss, adversarial loss, and total variational loss.

[0106] In one embodiment, the deep feature enhancement network includes a second shallow feature extraction module and a second deep feature extraction module connected in sequence. The second deep feature extraction module includes at least three coding layers, wherein the enhanced feature map is obtained based on the deep features output by the penultimate coding layer, the penultimate coding layer and the last coding layer in the second deep feature extraction module.

[0107] In one embodiment, the step of obtaining the pre-trained deep feature enhancement network includes: concatenating the deep feature network with the pre-trained head network; freezing the weights of the head network; training the deep feature network according to the total loss function for object detection; and obtaining the pre-trained deep feature enhancement network. The total loss function for object detection includes bounding box regression loss, object confidence loss, and category loss.

[0108] In one embodiment, the total loss function for object detection is:

[0109] L dt =λ5L obj +λ6L cls +λ7L CIoU

[0110] Among them, L dt Let L be the total loss function for object detection, where λ5, λ6, and λ7 are the weights of the object confidence loss, class loss, and bounding box regression loss, respectively. obj For target confidence loss, N is the total number of targets in the sample, t i c represents the IoU value between the ground truth bounding box and the predicted bounding box. i To predict confidence levels, L cls For category loss, y i For the true category label, p i L is the category prediction value. CIoU For bounding box regression loss, IoU is the crossover ratio, ρ 2 (b,b gt Let d be the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and let d be the diagonal distance of the smallest rectangular region that can simultaneously contain both the predicted and ground truth bounding boxes. w and h are the width and height of the predicted bounding box, respectively, and w and h are the width and height of the ground truth bounding box, respectively.

[0111] In one embodiment, the backbone network includes a first feature extraction module and three sets of second feature extraction modules connected in sequence. The first feature extraction module includes two coding layers and an L-ELAN module connected in sequence. The second feature extraction module includes an L-ELAN module and an MP module. The L-ELAN module includes a multi-scale convolution module, a multi-scale feature fusion module, and a fused feature extraction module. The multi-scale convolution module includes a first-scale convolution sub-module and a second-scale convolution sub-module. The first-scale convolution sub-module includes at least one coding layer. The second-scale convolution sub-module includes multiple coding layers with the same convolution kernel. The output features of each coding layer in the multi-scale convolution module are fused by the multi-scale feature fusion module and then used as the input of the fused feature extraction module. The fused feature extraction module extracts features to obtain the output features of the second feature extraction module.

[0112] In one embodiment, the augmentation detection fusion model is trained based on a pre-constructed loss function and low-quality infrared image samples to obtain a trained augmentation detection fusion model. This includes: freezing only the pre-trained weights of the shallow feature enhancement network and the deep feature enhancement network, and training the augmentation detection fusion model using the total target detection loss function and low-quality infrared image samples; freezing only the weights of the improved YOLOv7 target detection network, and training the augmentation detection fusion model using the total target detection loss function and low-quality infrared image samples to obtain a trained augmentation detection fusion model.

[0113] Specific limitations regarding the integrated infrared image multi-layer feature enhancement and detection device can be found in the limitations of the integrated infrared image multi-layer feature enhancement and detection method described above, and will not be repeated here. Each module in the aforementioned integrated infrared image multi-layer feature enhancement and detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0114] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an integrated method for infrared image multi-layer feature enhancement detection. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0115] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. An integrated method for multi-layer feature enhancement detection of infrared images, characterized in that, The method includes: Low-quality infrared image samples are acquired and input into an integrated enhancement and detection model. The integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network. The improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network. The low-quality infrared image sample is enhanced by the shallow feature enhancement network to obtain a high-quality infrared image. The high-quality infrared image is then enhanced by the deep feature enhancement network to obtain an enhanced feature map. The enhanced feature map is then convolved by the first convolutional layer to obtain a first feature map. The backbone network extracts multiple features from the high-quality infrared image to obtain a second feature map. The second feature map is then convolved by the second convolutional layer. The first feature map and the convolved second feature map are then fused by the stitching layer to obtain a fused feature map. The fused feature map is then predicted by the head network to obtain a prediction result. The prediction result is then displayed on the high-quality infrared image to output the target detection result. The enhanced detection integrated model is trained based on the pre-constructed loss function and the low-quality infrared image samples to obtain the trained enhanced detection integrated model. Infrared image target detection is performed using a trained integrated augmentation and detection model.

2. The method according to claim 1, characterized in that, The shallow feature enhancement network includes a generator network and a discriminator network, wherein the discriminator network is used to assist the training of the generator network, and the pre-trained generator network is used to enhance the features of the low-quality infrared image samples. The discriminator network comprises multiple coding layers, fully connected layers, and an output layer connected in sequence. The generator network includes a first shallow feature extraction module and a first deep feature extraction module connected in sequence. The first shallow feature extraction module and the first deep feature extraction module each include at least two coding layers. The shallow features output from the first coding layer in the first shallow feature extraction module are concatenated with the output features from the second-to-last coding layer in the first deep feature extraction module to obtain concatenated features. The concatenated features are then weighted and fused with the shallow features output from the second coding layer in the first shallow feature extraction module to obtain weighted fused features. The weighted fused features are then input into the last coding layer of the first deep feature extraction module to output a high-quality infrared image.

3. The method according to claim 2, characterized in that, The steps to obtain a pre-trained shallow feature enhancement network include: Obtain the high-quality infrared image sample corresponding to the low-quality infrared image sample; The high-quality infrared image output by the generator network and the high-quality infrared image sample are randomly input into the discriminator network for discrimination, and the corresponding discrimination result is output. The generator network is trained based on the total image enhancement loss function, the low-quality infrared image samples, the high-quality infrared image samples, and the discrimination results output by the discriminator network to obtain a pre-trained shallow feature enhancement network; the total image enhancement loss function includes pixel difference loss, adversarial loss, and total variational loss.

4. The method according to claim 1, characterized in that, The deep feature enhancement network includes a second shallow feature extraction module and a second deep feature extraction module connected in sequence. The second deep feature extraction module includes at least three encoding layers. The enhanced feature map is obtained based on the deep features output by the penultimate encoding layer, the second-to-last encoding layer, and the last encoding layer in the second deep feature extraction module.

5. The method according to claim 4, characterized in that, The steps to obtain a pre-trained deep feature enhancement network include: The deep feature network is concatenated with the pre-trained head network; The weights of the head network are frozen, and the deep feature network is trained according to the total loss function for object detection to obtain a pre-trained deep feature enhancement network; the total loss function for object detection includes bounding box regression loss, object confidence loss and category loss.

6. The method according to claim 5, characterized in that, The total loss function for target detection is: L dt = λ5L obj + λ6L cls + λ7L CIoU Among them, L dt Let L be the total loss function for object detection, where λ5, λ6, and λ7 are the weights of the object confidence loss, class loss, and bounding box regression loss, respectively. obj For target confidence loss, N is the total number of targets in the sample, t i c represents the IoU value between the ground truth bounding box and the predicted bounding box. i To predict confidence levels, L cls For category loss, y i For the true category label, p i L is the predicted value for the category. CIoU For bounding box regression loss, IoU is the crossover ratio, ρ 2 (b,b gt Let d be the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and let d be the diagonal distance of the smallest rectangular region that can simultaneously contain both the predicted and ground truth bounding boxes. w and h are the width and height of the predicted bounding box, respectively, and w and h are the width and height of the ground truth bounding box, respectively.

7. The method according to claim 1, characterized in that, The backbone network includes a first feature extraction module and three sets of second feature extraction modules connected in sequence; the first feature extraction module includes two coding layers and an L-ELAN module connected in sequence. The second feature extraction module includes an L-ELAN module and an MP module; the L-ELAN module includes a multi-scale convolution module, a multi-scale feature fusion module, and a fused feature extraction module. The multi-scale convolution module includes a first-scale convolution submodule and a second-scale convolution submodule. The first-scale convolution submodule includes at least one encoding layer; the second-scale convolution submodule includes multiple encoding layers with the same convolution kernel. The multi-scale feature fusion module fuses the output features of each coding layer in the multi-scale convolution module, and then uses them as input to the fusion feature extraction module. After feature extraction by the fusion feature extraction module, the output features of the second feature extraction module are obtained.

8. The method according to any one of claims 1-7, characterized in that, The augmentation detection integrated model is trained based on a pre-constructed loss function and low-quality infrared image samples to obtain a trained augmentation detection integrated model, which includes: Only the pre-trained weights of the shallow feature enhancement network and the deep feature enhancement network are frozen, and the integrated enhancement and detection model is trained using the total loss function for object detection and low-quality infrared image samples. Only the weights of the improved YOLOv7 object detection network are frozen, and the augmentation and detection integrated model is trained using the total object detection loss function and low-quality infrared image samples to obtain the trained augmentation and detection integrated model.

9. An integrated device for multi-layer feature enhancement and detection of infrared images, characterized in that, The device includes: The sample input module is used to acquire low-quality infrared image samples and input the low-quality infrared image samples into the integrated enhancement and detection model; the integrated enhancement and detection model includes a pre-trained shallow feature enhancement network, a pre-trained deep feature enhancement network, a first convolutional layer, and an improved YOLOv7 target detection network; the improved YOLOv7 target detection network includes a backbone network, a second convolutional layer, a stitching layer, and a head network. The feature enhancement module is used to enhance the features of the low-quality infrared image sample through the shallow feature enhancement network to obtain a high-quality infrared image, enhance the features of the high-quality infrared image through the deep feature enhancement network to obtain an enhanced feature map, and perform convolution processing on the enhanced feature map through the first convolutional layer to obtain a first feature map. The enhanced detection integrated module is used to extract multiple features from the high-quality infrared image through the backbone network to obtain a second feature map, perform convolution processing on the second feature map through the second convolutional layer, fuse the first feature map and the convolutional second feature map through the stitching layer to obtain a fused feature map, predict the fused feature map through the head network to obtain a prediction result, display the prediction result on the high-quality infrared image, and output the target detection result. The model training module is used to train the integrated enhancement detection model based on a pre-constructed loss function and the low-quality infrared image samples to obtain a trained integrated enhancement detection model. The target detection module is used to perform target detection in infrared images using a trained integrated augmented detection model.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images