Infrared weak and small target detection self-supervised learning method based on image degradation

Through self-supervised learning method based on image degradation, a pseudo-sample that meets the real environment and trains an infrared weak object detection model, the problems of high data acquisition cost and cumbersome labeling process in the existing technology are solved, and efficient infrared weak object detection is achieved .

CN119027772AActive Publication Date: 2024-11-26CHANGCHUN UNIV OF SCI & TECH

Patent Information

Application Number
CN202411484243.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-23
Publication Date
2024-11-26
Estimated Expiration
2044-10-23

AI Technical Summary

Technical Problem

The existing infrared weak target detection methods rely on manual annotation data, resulting in high data acquisition cost, cumbersome labeling process, single image background, uneven data distribution, and unable to effectively train high-precision weak target area segmentation models.

Method used

Using self-supervised learning methods based on image degradation, we can generate pseudo-samples that meet the real environment by constructing target degradation and background degradation models, and design an online pseudo-sample generation model, and generate realistic pseudo-sample images using global, local and grayscale constraints. , combines self-supervised learning framework and neural network for training, and directly uses the location-confidence tag for network training.

Benefits of technology

It reduces the computational complexity and time complexity, improves the accuracy of object detection, reduces the model size, improves the detection speed, and effectively solves the problems of uneven data distribution and imperfect labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027772A_ABST
    Figure CN119027772A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of target detection, and particularly relates to an infrared weak and small target detection self-supervised learning method based on image degradation, and the method specifically comprises the following steps: S1, preparing data sets: preparing six data sets: a data set 1, a data set 2, a data set 3 and a data set 6 for pseudo sample generation; according to the infrared weak and small target detection self-supervised learning method based on image degradation, a novel degradation model fusing fuzzy degradation, brightness / contrast degradation and scale degradation modes is constructed, and an online gridding high-order constraint pseudo sample generation model combining global target constraint, local target constraint and local gray level constraint is designed. A new position-confidence # imgabs0 # evaluation index is provided, a simple and efficient neural network is designed, a complete network self-supervised training framework is closed to enable the network to converge successfully, the detection precision and the detection speed of a weak and small target are improved, and the problems that calculation complexity is too high and data are difficult to obtain are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of target detection, and in particular to a self-supervised learning method for infrared dim small target detection based on image degradation. Background Art

[0002] Infrared dim small target detection is an important part of image processing tasks. It aims to identify the position and pixel information of dim small targets in infrared images. It has been successfully applied in the fields of ground-to-air drone detection, long-distance ship identification at sea, and foreign object intrusion detection in complex scenes.

[0003] In recent years, although the infrared weak target detection task has developed very maturely, the fully supervised detection method has almost only focused on the design of the network structure and the accuracy of the weak target segmentation area ratio. In fact, to train a high-precision weak target area segmentation model, it depends on large-scale image data and perfect binary labels. Even with a large-scale image dataset, the labeling of binary labels is time-consuming and labor-intensive. The existing method of manually shooting real weak target images and then labeling them highlights three key problems at the data level: 1) Single image background: Due to the complexity of the acquisition work, the manually collected image scenes are too single, and each detection method can usually only be generalized on one dataset. 2) Unbalanced data distribution: In the manually collected single target images, the size of the weak target is usually between 3×3 and 6×6, resulting in a serious imbalance in the distribution of image background and target information, while the cost of multi-target image acquisition is high and multiple targets cannot be completely randomly distributed in the field of view of a single lens. 3) Imperfect annotation: Even if the existing weak target binary labels generated by pseudo samples have been successfully labeled, pixel-level labeling cannot be achieved, which is one of the important reasons for the accuracy defects of image segmentation-based methods. Therefore, it is very necessary to design an infrared dim small target detection method based on a self-supervised learning framework.

[0004] As discussed above, the small targets in infrared images present distressing characteristics such as small size, no features and uncertain shape. The fully supervised detection method designed only to improve the segmentation area of ​​small targets cannot fully train the neural network to truly understand the structural information of the image. Therefore, the present invention believes that the key to detecting small targets in infrared images is to determine the position information of the small targets. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a self-supervised learning method for infrared dim target detection based on image degradation, which solves the problems of extremely unbalanced distribution of target information and background information samples, unreasonable layout and unrealistic samples in image data generated by existing pseudo samples due to the high cost of image data acquisition and the cumbersome and time-consuming labeling process. It solves the problem of high computational complexity and time complexity caused by a large number of image-level calculations in the fully supervised learning method.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A self-supervised learning method for infrared dim small target detection based on image degradation specifically comprises the following steps:

[0010] S1, prepare datasets: prepare six datasets, dataset 1, dataset 2, dataset 3 and dataset 6 for pseudo sample generation; dataset 4 and dataset 5 are used for the final model testing;

[0011] S2, obtain infrared target and background samples: obtain target positive samples for pseudo sample generation through simulation generation method and threshold clipping method; obtain background negative samples for pseudo sample generation by mixing the public infrared background data set in step S1;

[0012] S3, constructing an image degradation model: the image degradation model includes a target degradation process and a background degradation process; inputting the target positive sample and the background negative sample obtained in step S2 into the image degradation model to obtain a degraded sample library;

[0013] S4, build an online pseudo sample generation model: the online pseudo sample generation model mainly includes global target constraints, local target constraints and local grayscale constraints; the degraded sample library obtained in S3 is input into the pseudo sample generation model to generate infrared weak target pseudo sample images and (Location-Confidence Label) dataset;

[0014] S5, build a self-supervised learning framework: The self-supervised learning framework mainly includes the design of neural networks and training strategies;

[0015] S6, selection of loss function and design of evaluation index: select a suitable loss function kernel and design position-confidence loss; the loss function kernel is mean square error (MSE); design evaluation indexes to evaluate the performance and accuracy of the detection model, including designing the position deviation rate ( ) indicator, adaptation detection rate ( ) and false alarm rate ( )index.

[0016] S7, model training and fine-tuning: training the infrared small target detection model, training the pseudo sample image and position-confidence label obtained in step S4 in the self-supervised learning framework built in step S5; using the real data set prepared in step S1 to fine-tune the model, optimize the model parameters, and further improve the performance of small target detection;

[0017] S8, model saving and testing: After the training and fine-tuning in step S7 are completed, the fine-tuned network parameters are solidified and the final small target detection model is saved; the model is tested using the remaining real data set in step S1, and the real infrared image can be directly input into the trained end-to-end model to obtain the result image marked with the detection box and position information.

[0018] Furthermore, in said S1, three public infrared background datasets, namely, FLIR dataset, TNO dataset, and MSRS dataset, are used to obtain a weak target sample database by threshold clipping; a self-made infrared dataset containing weak targets, namely, NUAA-SIRST dataset and IRDST-real dataset, are used to obtain a real target sample database; two public real datasets containing weak targets, namely, NUAA-SIRST dataset and IRDST-real dataset, are used for the final model test;

[0019] Furthermore, in S3, the image degradation model is decoupled into the target degradation process and the background degradation process to analyze the optical imaging characteristics of the target and the background respectively; the degradation function library of the two is composed of complex degradation factors such as blur degradation, brightness / contrast degradation and scale degradation, so as to fit the image that is more in line with the degradation mechanism of the real environment;

[0020] The blur degradation factor is composed of anisotropic Gaussian blur functions and multi-directional motion blur functions, and is used to suppress blur degradation of both the background and the target of a real infrared weak target image, which is caused by the imaging system and the target always being in a state of relative motion.

[0021] The brightness / contrast degradation factor is used to suppress the brightness / contrast degradation of the background sample image due to the influence of low illumination, high brightness and smoke, and the brightness / contrast degradation of the target sample image due to the influence of imaging distance and light environment factors, resulting in different levels of brightness / contrast degradation;

[0022] The scale degradation factor is composed of a random up-and-down sampling function and a random attitude rotation function, and is used to suppress scale degradation of background samples caused by random adjustments of different cameras and camera resolutions.

[0023] Furthermore, in S4, the global target constraint includes a quantity constraint and a position constraint, which ensures that the number of targets added to the overall infrared background image is within a reasonable range, and the overall position layout of the constrained targets added to the background image is more reasonable;

[0024] Furthermore, in S4, the local target constraints include image gridding and quantity constraints, which cleverly avoid the problem of uneven target distribution in the local area and eliminate the "pseudo target" phenomenon.

[0025] Furthermore, in S4, the local grayscale constraint includes a local standard deviation index and target elimination, which solves the problem that the randomly added target and the local background area feature distribution are extremely similar due to the target sample suffering the most severe degradation.

[0026] Furthermore, in S4, the online pseudo sample generation model output interface includes a pseudo sample image output interface, a binary label image output interface and (Position-Confidence Label) sample output interface for subsequent network training and image processing tasks;

[0027] Furthermore, in S5, the neural network is a simple and efficient residual network, including a head network, a feature extraction network and a tail network, which is used to extract deep features of the image, stabilize the reverse conduction of the gradient, and connect the output of the network to the (position-confidence label) for matching.

[0028] The head network, consisting of a convolutional layer, a normalization layer and a maximum pooling layer, is used to encode the image into a feature domain;

[0029] The feature extraction network is composed of multiple residual blocks, each of which is composed of a convolutional layer, a normalization layer (BN) and an activation function (ReLu), which is used to extract deep feature information of the image and focus on the target position in the infrared image;

[0030] The tail network is composed of an average pooling layer, a fully connected layer and an activation function (Sigmoid is used for position activation and Tanh is used for confidence activation), which is used to output image features adapted to network training;

[0031] Furthermore, in S6, the position-confidence loss includes position loss and confidence loss, which is used to utilize The position information and confidence information in the network are respectively used to calculate the loss value with the position and confidence output by the network.

[0032] (III) Beneficial effects

[0033] Compared with the prior art, the present invention provides a self-supervised learning method for infrared dim small target detection based on image degradation, which has the following beneficial effects:

[0034] The present invention designs an online pseudo sample generation method based on image degradation. First, an image degradation model consisting of target degradation and background degradation is designed to obtain a target and background sample database that conforms to the real environment degradation; and an online gridded high-order constrained pseudo sample generation model is designed. Through three constraint approaches, the position, quantity and authenticity of small targets are constrained to generate pseudo sample images, bridging the information gap between pseudo samples and real samples.

[0035] This paper constructs a self-supervised learning framework, which for the first time combines online target position-confidence array labels in the field of infrared dim small target detection. As the only criterion for constraining network training, it has been successful. This training method is a direct pixel-level target positioning method, which effectively improves the accuracy of target detection and reduces the computational complexity. Experiments have shown that the computational complexity and time complexity of the infrared dim target detection method proposed in this invention have decreased by about 9.4% and 11.1% respectively, the model size is only 1.4M, and the detection speed has also been greatly improved.

[0036] The present invention designs a position deviation rate evaluation index ( ), by calculating the correctly detected target The center of gravity position deviation is measured to evaluate the accuracy of the target position, and a large number of experiments are carried out to verify the significant advantages of the training model proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a self-supervised learning method for infrared dim small target detection based on image degradation;

[0038] Figure 2 A schematic diagram of a method for constructing a background negative sample database and a target positive sample database according to the present invention;

[0039] Figure 3 This is a working principle diagram of the image degradation model described in the present invention;

[0040] Figure 4 A schematic diagram of a high-order constraint method for generating online pseudo samples according to the present invention;

[0041] Figure 5 A neural network and training strategy framework diagram for self-supervised learning according to the present invention;

[0042] Figure 6 This is a qualitative comparison effect diagram of the infrared dim small target detection method described in this invention and the existing method;

[0043] Figure 7 The figure is a schematic diagram for quantitatively comparing the infrared dim small target detection method described in the present invention with the existing methods. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1:

[0046] like Figure 1-7 As shown, a flowchart of a self-supervised learning method for infrared dim small target detection based on image degradation proposed by an embodiment of the present invention, the method specifically comprises the following steps:

[0047] S1, prepare data sets: prepare three public infrared background data sets, namely FLIR data set 1, TNO data set 2 and MSRS data set 3, use the random cropping method to expand the background data set, and use the sample elimination method to eliminate the unbalanced background for pseudo sample generation; prepare a self-made real data set 6 containing weak targets to obtain real samples of weak targets; prepare NUAA-SIRST real data set 4 and IRDST-real real data set 5 for model testing;

[0048] S2, obtain infrared target and background samples: obtain background negative samples for pseudo sample generation by mixing the public infrared background data set in step S1; obtain target positive samples for pseudo sample generation by using the simulated generation method and threshold clipping of the self-made real data set;

[0049] S3, build image degradation model: build image degradation model including target degradation process and background degradation process; before degradation, first process background and target samples by sample shuffle (Degradation shuffle), and combine the overall degradation process; target and background degradation process are uniformly composed of blur degradation, brightness / contrast degradation and scale degradation; blur degradation is composed of heterogeneous Gaussian blur and multi-directional motion blur; for target sample images, use smaller random heterogeneous Gaussian blur kernel and standard deviation to avoid excessive blurring and causing target disappearance, and use larger blur kernel and standard deviation for background sample images. Heterogeneous Gaussian blur has a negative effect on the input original image. Perform convolution operation with blur kernel to generate anisotropic Gaussian blurred degraded image ; Multi-directional motion blur must first ensure the randomness of the blur kernel, and then design random blur operators in the horizontal direction, vertical direction, and arbitrary direction of the kernel. and , for the anisotropic Gaussian blurred degraded image Convolution operation with blur operator to obtain multi-directional motion blurred degraded image , multi-directional motion blur simulates the more complex motion blur degradation of small infrared targets in the real world. Brightness / contrast degradation, by adjusting the factor Learning random background and target sample brightness / contrast degradation distribution probability and multi-directional motion blurred degraded images Perform multiplication to obtain brightness / contrast degraded image ; Scale degradation consists of random up-sampling and random posture rotation; Random up-sampling methods include nearest neighbor interpolation, bilinear interpolation and bicubic interpolation; The sampling probabilities of the three sampling methods are determined by the probability distribution learned by the model, which is used to achieve data enhancement and obtain randomly up-sampled degraded images The random pose rotation function is composed of the random rotation factor Determine the random rotation angle to be Range, after random posture rotation, the final degraded image is obtained The acquired target sample degraded image and background sample degraded image are used for pseudo sample generation.

[0050] S4, building an online pseudo sample generation model: It is unrealistic to use the original undegraded background samples and target samples to directly drive the network to learn the data distribution. The process of directly synthesizing pseudo sample data from degraded image samples requires adding complex constraints to perfectly simulate the imaging mechanism of the real environment. Based on this, the present invention designs an online gridding high-order constrained pseudo sample generation method, which generates realistic pseudo samples through global target constraints, local target constraints and local grayscale constraints; the present invention imposes quantity constraints and position constraints in the global target constraints; first, the number of targets added in the infrared global background is constrained to be within a reasonable range, ensuring that the target quantity in the generated sample is within a specific range to maintain the consistency and rationality of the sample target quantity; secondly, for an image in a certain specific data set, the target adding position is set according to the distribution characteristics of the image information. Specifically, if the content of the image in the data set is relatively simple, for example, the image content is mainly composed of the sky, we can add targets in any area. If the content of the image in the data set is relatively complex, for example, the image contains elements such as the sky, buildings, trees and pedestrians, the present invention is more inclined to set the adding position of most targets in the upper part of the image, and a few targets are added in other areas of the complex content; in order to generate a more realistic infrared weak target image, the present invention adds local constraints by a gridding method. First, the image resolution is unified to 512×512, and then divided into 64 image blocks with a resolution of 64×64 to form a gridded layout in the global image. Then, the number of targets in each image block is constrained to be within a reasonable range. The local target constraint cleverly avoids the problem of uneven target distribution in local areas and eliminates the "pseudo target" phenomenon. The target sample may suffer the most serious degradation after the degradation model, causing the visibility of the target to be close to disappearance. Furthermore, the global target constraint also fails to ensure that all target samples are generated at reasonable positions of background samples. If a target is randomly generated in an area that is extremely similar to its own feature distribution, this target must be eliminated. Based on this, the present invention defines a tolerable local standard deviation change threshold and uses local grayscale to determine the rationality of target generation on a local image block.

[0051] S5, build a self-supervised learning framework: the self-supervised learning framework mainly includes the design of neural networks and the design of training strategies; in the training stage, a simple neural network is trained using the simulated data set obtained in this article, and in the testing stage, the trained model is tested on a real infrared weak target data set. Therefore, the present invention designs a simple and efficient residual neural network for training, stabilizes the reverse conduction of the gradient, and matches the output of the network with the result; our residual network consists of a head network, a feature extraction network, and a tail network. Among them, the feature extraction network is composed of residual blocks combined into 3, 6, and 9 numbers, and each residual block is composed of the most basic residual network; the present invention directly supervises network training from the perspective of target positioning to achieve high-precision detection of the target. During the training phase, we work in an end-to-end mode. The original infrared background samples and target samples undergo image degradation and pseudo-sample generation to generate pseudo-sample data containing infrared small target sample images, mask binary labels, and position-confidence labels. The mask binary labels are used for subsequent image processing tasks, and the infrared small target sample images are sent to the neural network. Under the action of the loss function, the network output is continuously constrained to fit the optimal model with the position-confidence label.

[0052] S6, selection of loss function and design of evaluation index: select a suitable loss function kernel and design position-confidence loss; the loss function kernel is mean square error (MSE); design evaluation indexes to evaluate the performance and accuracy of the detection model, including designing the position deviation rate ( ) indicator, using the network output and the location information of the location-confidence label to calculate the deviation rate; the adaptation detection rate ( ) and false alarm rate ( )index, It represents the ratio of the number of correct targets detected to the total number of true targets in the label. Indicates the ratio of the number of detected wrong targets to the total number of targets in the label.

[0053] S7, model training and fine-tuning: training the infrared small target detection model, training the pseudo sample image and position-confidence label obtained in step S4 in the self-supervised learning framework built in step S5; using the real data set prepared in step S1 to fine-tune the model, optimize the model parameters, and further improve the performance of small target detection;

[0054] S8, model saving and testing: After the training and fine-tuning in step S7 are completed, the fine-tuned network parameters are solidified and the final small target detection model is saved; the model is tested using the remaining real data set in step S1, and the real infrared image can be directly input into the trained end-to-end model to obtain the result image marked with the detection box and position information.

[0055] Embodiment 2:

[0056] like Figure 1 As shown, a self-supervised learning method for infrared dim small target detection based on image degradation is provided, and the method specifically comprises the following steps:

[0057] S1, prepare data sets, prepare three public infrared background data sets, and shuffle the three data sets to enhance data randomness; data set one is the FLIR data set, the present invention pre-selects 4694 infrared images from the FLIR data set, because there are a large number of images with extremely high background similarity in the FLIR data set, so when selecting image data, try to ensure the richness of the image background; data set two is the TNO data set, the present invention pre-selects 710 infrared images from the TNO data set, because TNO contains infrared background images in harsh environments, so the generated pseudo sample image data will enhance the generalization ability of the model; data set three is the MSRS data set, the present invention pre-selects 3255 infrared images from the TNO data set, the MSRS data set contains night infrared images, which can improve the night target detection performance; Figure 2 (d) and Figure 2 As shown in (e), a total of 8659 background image data are prepared, the background data set is expanded using the random cropping method, and the unbalanced background is eliminated using the sample elimination method, resulting in 21419 final infrared background images for pseudo sample generation; Figure 2 As shown in (a), a self-made real dataset 6 containing small targets is prepared to obtain real samples of small targets. The dataset 6 contains 340 clear real image data of small UAV multi-target targets; two public real infrared small target datasets are prepared for model testing. NUAA-SIRST real dataset 4 and IRDST-real real dataset 5 are selected with 100 images each. It is necessary to ensure that the number, location and types of targets in the selected test datasets are rich.

[0058] S2, obtain infrared target and background samples: mix the public infrared background data set in step S1 and perform data enhancement to obtain 21419 background negative samples for pseudo sample generation; generate 1660 binary weak target images of 30×30 pixels in total for pseudo sample generation through simulation generation method; Figure 2 (b) and Figure 2 As shown in (c), 1269 binary weak target images are obtained by threshold cropping the self-made real data set for pseudo sample generation;

[0059] S3, constructing image degradation model: Figure 3As shown in the figure, an image degradation model including target degradation process and background degradation process is constructed; before degradation, the background and target samples are first processed by sample shuffle (Degradation shuffle) to combine the overall degradation process; the target and background degradation processes are uniformly composed of blur degradation, brightness / contrast degradation and scale degradation; blur degradation is composed of heterogeneous Gaussian blur and multi-directional motion blur; for the target sample image, a smaller random heterogeneous Gaussian blur kernel and standard deviation are used to avoid excessive blurring and causing the target to disappear, and for the background sample image, a larger blur kernel and standard deviation are used. The heterogeneous Gaussian blur has a certain effect on the input original image. Perform convolution operation with blur kernel to generate anisotropic Gaussian blurred degraded image , random anisotropic Gaussian blur is expressed as the following process using the formula:

[0060] (1)

[0061] in, represents the input image; Represents a 2D convolution operation; The Gaussian kernel representing the random kernel size and random standard deviation can be expressed using the following formula:

[0062] (2)

[0063] in, , , and Represent the width, height, Directional standard deviation and Directional standard deviation, , , , ;in, and , and , and , and Respectively represent the minimum width and maximum width of the Gaussian kernel, the minimum height and maximum height of the Gaussian kernel, and the calculation The minimum and maximum seeds required for the directional standard deviation random function, calculation The minimum and maximum seeds required for the directional standard deviation random function; in background degradation, , , , ; In target degradation, set , , , ; represents the random seed generation function, and The minimum and maximum ranges for generating seed numbers;

[0064] Multi-directional motion blur must first ensure the randomness of the blur kernel, and then design random blur operators in the horizontal direction, vertical direction, and arbitrary direction of the kernel. and , for the anisotropic Gaussian blurred degraded image Convolution operation with blur operator to obtain multi-directional motion blurred degraded image Multi-directional motion blur simulates the more complex motion blur degradation of infrared dim targets in the real world. Multi-directional motion blur can be expressed as:

[0065] (3)

[0066] in, and They represent the kernel horizontal and kernel vertical blur operators respectively; Indicates the size of the blur kernel, usually an odd number to ensure that the blur kernel has a central pixel; Indicates the position index of the current pixel; Indicates the random angle value generated to generate a random blur kernel, the range is ; This is a radian operation.

[0067] Brightness / contrast degradation, by adjusting the factor Learning random background and target sample brightness / contrast degradation distribution probability and multi-directional motion blurred degraded images Perform multiplication to obtain brightness / contrast degraded image , the brightness / contrast degradation function is expressed as:

[0068] (4)

[0069] in, Represents the brightness / contrast degradation factor, the background sample is degraded , the target sample is degraded .

[0070] Scale degradation consists of random up-sampling and random posture rotation; random up-sampling methods include nearest neighbor interpolation, bilinear interpolation and bicubic interpolation; the sampling probabilities of the three sampling methods are determined by the probability distribution learned by the model, which is used to achieve data enhancement and obtain randomly up-sampled degraded images The random pose rotation function is composed of the random rotation factor Determine the random rotation angle to be Range, after random posture rotation, the final degraded image is obtained , the present invention creates a sample scale degradation function, which can be expressed as follows:

[0071] (5)

[0072] in, is a random up-and-down sampling function, are random sampling factors of 2, 3, and 4, The random sampling methods are nearest neighbor, bilinear and bicubic; is the random posture rotation function, Indicated in Random rotation factor for range, not included in background sample degradation ;

[0073] The acquired target sample degraded image and background sample degraded image are used for pseudo sample generation.

[0074] S4, build an online pseudo sample generation model: Figure 4 As shown in the figure, the present invention designs an online grid-based high-order constrained pseudo sample generation method, which generates realistic pseudo samples through global target constraints, local target constraints and local grayscale constraints; the present invention imposes quantity constraints and position constraints in the global target constraints; first, constrains the number of targets added to the infrared global background In a reasonable range To ensure that the target quantity in the generated sample is within a specific range, so as to maintain the consistency and rationality of the sample target quantity; secondly, for a certain data set with a size of Specifically, if the image content in the dataset is relatively simple, for example, the image content is mainly composed of the sky, we can add targets in any area ( and , Indicates If the content of the image in the dataset is relatively complex, for example, the image contains elements such as the sky, buildings, trees, and pedestrians, the present invention tends to set the location where most objects are added to the upper part of the image, and a few objects are added to other areas of the complex content, which can be expressed by the following formula:

[0075] (6)

[0076] in, is an indicator function, which takes the value 1 when the condition is met and 0 otherwise; is the scale parameter, Indicates the ratio of adding objects to the upper part of the image.

[0077] In order to generate more realistic infrared weak target images, the present invention adds local constraints through a gridding method. First, the image resolution is The image is unified to 512×512 size and then divided into 64 blocks (8 blocks×8 blocks) with a size of 64×64 image blocks , forming a grid layout in the global image, the process is expressed as:

[0078] (7)

[0079] Furthermore, formula (7) reflects Represents the image block slicing operation. The present invention uses Indicates the row and column index of the image block position, and uses slicing to obtain the first Line The column is the center of the image patch with a size of 64×64; and Indicates the starting and ending index of the row; and Indicates the starting and ending index of the column;

[0080] Then, constrain each image patch The number of targets in In a reasonable range In the local target constraint, the problem of uneven target distribution in the local area is cleverly avoided, and the "pseudo target" phenomenon is also eliminated;

[0081] The target sample may suffer the most severe degradation after the degradation model, causing the visibility of the target to be close to disappearance; furthermore, the global target constraint also fails to ensure that all target samples are generated at reasonable positions of background samples. If the target is randomly generated in an area that is extremely similar to its own feature distribution, then this target must be eliminated. Based on this, the present invention defines the target before and after generation. The gray value is and , is the pixel index in the image block, and is defined is the tolerable local standard deviation change threshold, and the local image block is determined by the local gray level. The rationality of the above target generation can be expressed by the following formula:

[0082] (8)

[0083] In formula (8), and Respectively represent the target before and after generation The local standard deviation of and Indicates before and after target generation The local average gray value. If it satisfies ( Indicates that the image block index is ( ) local standard deviation before and after target generation), the present invention believes that generating targets under global constraints and local constraints is the most realistic.

[0084] S5, build a self-supervised learning framework: The self-supervised learning framework mainly includes the design of the neural network and the design of the training strategy; in the training phase, the simulated data set obtained in this paper is used to train a simple neural network, and in the testing phase, the trained model is used to test on a real infrared weak target data set. Therefore, the present invention designs a simple and efficient residual neural network for training, stabilizes the reverse conduction of the gradient, and matches the output of the network with the test, such as Figure 5 As shown in (b); our residual network consists of a head network, a feature extraction network and a tail network. The feature extraction network consists of 3, 6, and 9 residual blocks, each of which is composed of the most basic residual network; Figure 5 As shown in (a), in the training stage, our working mode is end-to-end mode. The original infrared background samples and target samples are subjected to image degradation and pseudo sample generation to generate infrared weak target sample images, mask binary labels and position-confidence labels as pseudo sample data. The mask binary labels are used for subsequent image processing tasks. The infrared weak target sample images are sent to Figure 5 (b) In the neural network, the loss function continuously constrains the network output to fit the optimal model of position-confidence labels.

[0085] S6, selection of loss function and design of evaluation index: select a suitable loss function kernel and design position-confidence loss; the loss function kernel is mean square error (MSE); design evaluation indexes to evaluate the performance and accuracy of the detection model, including designing the position deviation rate ( ) indicator, using the network output and the location information of the location-confidence label to calculate the deviation rate, The formula used is as follows:

[0086] (9)

[0087] Adaptation detection rate ( ) and false alarm rate ( )index, It represents the ratio of the number of correct targets detected to the total number of true targets in the label. The calculation formula is:

[0088] (10)

[0089] It represents the ratio of the number of detected wrong targets to the total number of targets in the label. The calculation formula is:

[0090] (11)

[0091] It should be noted that we set the statistics When , the confidence in the model output is guaranteed to be greater than 0.7 and the position error is less than 30 pixels; Statistics This ensures that the confidence in the model output is less than 0.7 and the position error is greater than 30 pixels.

[0092] S7, model training and fine-tuning: training the infrared small target detection model, training the pseudo sample image and position-confidence label obtained in step S4 in the self-supervised learning framework built in step S5; using the real data set prepared in step S1 to fine-tune the model, optimize the model parameters, and further improve the performance of small target detection;

[0093] All software programming parts of the present invention are carried out on the open source Pytorch framework, and hardware training, testing, model tuning and later program modularization are all carried out on a Nvidia GeForce RTX4090 GPU computer. The loss function kernel of the present invention uses MSE loss, and in order to improve the detection accuracy, the position and confidence use MSE loss separately, and the weights of the position loss and the confidence loss are respectively , During network training, the number of multi-core processes is set to 4, the learning rate is set to 0.01, the SGD gradient descent method is used to constrain network optimization, the batch size is set to 64, and the network achieves stable convergence after 200 epochs.

[0094] S8, model saving and testing: After the training and fine-tuning in step S7 are completed, the fine-tuned network parameters are solidified and the final small target detection model is saved; the model is tested using the remaining real data set in step S1, and the real infrared image can be directly input into the trained end-to-end model to obtain the result image marked with the detection box and position information.

[0095] Among them, the implementation of convolution, normalization, activation function, sample cropping and sample elimination are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0096] The present invention constructs a self-supervised learning method for infrared small target detection based on image degradation, thus getting rid of the dilemma of existing fully supervised learning methods that rely on training on real infrared data sets that are difficult to annotate. As far as we know, the method designed by the present invention is one of the few methods in the field of infrared small target detection that successfully trains the network and achieves stable convergence by combining original infrared images with position-confidence labels; the self-supervised learning framework proposed in the present invention can not only effectively suppress the complex degradation effects of real imaging and improve detection efficiency, but also save infrared small target detection in deep learning applications due to data-driven difficulties. Qualitative comparison of target detection in two real data sets five and six between the prior art and the method proposed in the present invention. Figure 6 As shown;

[0097] Under the same conditions, the feasibility and superiority of this method are further verified by calculating the relevant indicators of infrared dim target detection results obtained by the existing method. Figure 7 As shown in the figure, it can be seen that the method proposed in the present invention has a higher detection rate, lower position deviation rate and false alarm rate, and the trained model is only 1.4M; in the test phase, the computational complexity and time complexity of the infrared weak target detection method proposed in the present invention decreased by about 9.4% and 11.1% percentage points respectively.

[0098] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A self-supervised learning method for infrared dim small target detection based on image degradation, characterized by: The method specifically comprises the following steps: S1, prepare datasets: prepare six datasets, dataset 1, dataset 2, dataset 3 and dataset 6 for pseudo sample generation; dataset 4 and dataset 5 are used for the final model testing; S2, obtain infrared target and background samples: obtain target positive samples for pseudo sample generation through simulation generation method and threshold clipping method; obtain background negative samples for pseudo sample generation by mixing the public infrared background data set in step S1; S3, constructing an image degradation model: the image degradation model includes a target degradation process and a background degradation process; inputting the target positive sample and the background negative sample obtained in step S2 into the image degradation model to obtain a degraded sample library; S4, build an online pseudo sample generation model: The online pseudo sample generation model mainly includes global target constraints, local target constraints and local grayscale constraints; the degraded sample library obtained in S3 is input into the pseudo sample generation model to generate infrared weak target pseudo sample images and position-confidence labels Datasets; S5, build a self-supervised learning framework: The self-supervised learning framework mainly includes the design of neural networks and training strategies; S6, selection of loss function and design of evaluation index: select the appropriate loss function kernel and design the position-confidence loss; The kernel of the loss function is the mean square error function MSE; the evaluation indicators are designed to evaluate the performance and accuracy of the detection model, including the design position deviation rate Indicator, Adaptation Detection Rate and false alarm rate index; S7, model training and fine-tuning: training the infrared dim small target detection model, and training the pseudo sample image and position-confidence label obtained in step S4 in the self-supervised learning framework built in step S5; Use the real data set prepared in step S1 to fine-tune the model and optimize the model parameters to further improve the performance of small target detection; S8, model saving and testing: after the training and fine-tuning in step S7 are completed, the fine-tuned network parameters are solidified and the final small target detection model is saved; Using the remaining real data set in step S1 to test the model, the real infrared image can be directly input into the trained end-to-end model to obtain a result image annotated with the detection box and position information.

2. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized in that: In S1, three public infrared background datasets, namely, FLIR dataset, TNO dataset, and MSRS dataset, are used to obtain a small target sample database by threshold clipping; a self-made infrared dataset containing small targets is used to obtain a real target sample database; two public real datasets containing small targets, namely, NUAA-SIRST dataset and IRDST-real dataset, are used for the final model test.

3. The infrared dim small target detection self-supervised learning method based on image degradation according to claim 1 is characterized by: In S3, the image degradation model is decoupled into the target degradation process and the background degradation process to analyze the optical imaging characteristics of the target and the background respectively; the degradation function library of the two is composed of complex degradation factors such as blur degradation, brightness / contrast degradation and scale degradation, so as to fit the image that is more in line with the degradation mechanism of the real environment; The blur degradation factor is composed of anisotropic Gaussian blur function and multi-directional motion blur function, and is used to suppress blur degradation of both the background and the target of the real infrared weak target image due to the imaging system and the target being in a relative motion state at all times; The brightness / contrast degradation factor is used to suppress the brightness / contrast degradation of the background sample image due to the influence of low illumination, high brightness and smoke, and the brightness / contrast degradation of the target sample image due to the influence of imaging distance and light environment factors, resulting in different levels of brightness / contrast degradation; The scale degradation factor, which is composed of a random up-and-down sampling function and a random attitude rotation function, is used to suppress scale degradation of background samples caused by random adjustments of different cameras and camera resolutions.

4. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized in that: In S4, the global target constraint includes quantity constraint and position constraint, which ensures that the number of targets added in the overall infrared background image is within a reasonable range and the overall position layout of the constrained targets added in the background image is more reasonable.

5. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized in that: In S4, the local target constraints include image gridding and quantity constraints, which cleverly avoid the problem of uneven target distribution in the local area and eliminate the "pseudo target" phenomenon.

6. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized in that: In S4, the local grayscale constraint includes local standard deviation index and target elimination, which solves the problem that the randomly added targets and the local background area feature distribution are extremely similar due to the target samples suffering the most severe degradation.

7. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized in that: In S4, the online pseudo sample generation model output interface includes a pseudo sample image output interface, a binary label image output interface and Sample output interface for subsequent network training and image processing tasks.

8. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1 is characterized by: In S5, the neural network is a simple and efficient residual network, including a head network, a feature extraction network and a tail network, which is used to extract deep features of the image, stabilize the reverse conduction of the gradient, and compare the output of the network with Perform anastomosis; The head network, consisting of a convolutional layer, a normalization layer and a maximum pooling layer, is used to encode the image into a feature domain; The feature extraction network is composed of multiple residual blocks, each of which is composed of a convolutional layer, a normalization layer and a ReLu activation function, and is used to extract deep feature information of the image and focus on the target position in the infrared image; The tail network consists of an average pooling layer, a fully connected layer and an activation function. The position activation uses a Sigmoid activation function, and the confidence activation uses a Tanh activation function, which is used to output image features adapted to network training.

9. The self-supervised learning method for infrared dim small target detection based on image degradation according to claim 1, characterized in that: In S6, the position-confidence loss includes position loss and confidence loss, which is used to utilize The position information and confidence information in the network are respectively used to calculate the loss value with the position and confidence output by the network.

Citation Information

Patent Citations

  • Small target image detection method based on conditional generative adversarial and dense connection network

    CN116824327A

  • Hierarchical gating cross Transform infrared dim target detection method

    CN116912660A

  • High-frame-rate and high-precision detection method for aerial infrared miniature target

    CN118691956A

  • System and method for estimating number and range of a plurality of moving targets

    US20170254893A1

  • Semi-weakly supervised object detection using progressive knowledge transfer and pseudo-label mining

    WO2023139407A1

Cited By

  • Low-slow small moving target sample generation method based on degradation model

    CN120339752A

  • Target detection and identification method under complex background

    CN120339848A

  • Pre-screening method and system for rail transit engineering video image quality degradation

    CN120655581A

  • Rail transit engineering video image quality degradation-oriented pre-screening method and system

    CN120655581B

  • Infrared target detection method and device for self-supervised learning, equipment and storage medium

    CN120953574A