A method for detecting and locating centroid of dim small target in large-size infrared image
By combining random image preprocessing with an improved U-net network and loss function, the problems of low efficiency and poor robustness in detecting small targets with large size and low signal-to-noise ratio in infrared images are solved, achieving high-precision centroid localization and reducing computational complexity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
- Filing Date
- 2023-04-04
- Publication Date
- 2026-06-02
AI Technical Summary
Existing infrared image target detection methods are inefficient and have poor robustness when dealing with large targets and low signal-to-noise ratios, making it difficult to accurately and quickly detect small targets and locate their centroids, which increases the complexity of the algorithms.
We employ random image preprocessing, improve the U-net network design to create a multi-branch feature extraction network with an attention mechanism, and combine Dice Loss and Focal Loss loss functions to directly locate the centroid, thereby reducing the impact of noise and improving the accuracy of feature extraction.
This improved the accuracy of centroid localization to the sub-pixel level, reduced computational load and algorithm complexity, and enhanced robustness.
Smart Images
Figure CN116363347B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting and locating the centroid of small, faint targets in large-size infrared images. Background Technology
[0002] Infrared images typically have over 100 million pixels (around 1000 pixels * 1000 pixels), while the target size is very small (<20 pixels * 20 pixels). Using conventional whole-image training methods is slow and inefficient. To accurately and quickly detect and locate the centroid of small targets (<20 pixels * 20 pixels) in large (around 1000 pixels * 1000 pixels) infrared images with low signal-to-noise ratios, most current infrared small target detection methods detect the target's bounding box. To obtain the target's centroid location information, a secondary calculation is required, increasing the algorithm's complexity and reducing robustness, making it unsuitable for practical applications. Summary of the Invention
[0003] In view of the above problems, the present invention provides a method for detecting and locating the centroid of small, dark targets in large-size infrared images.
[0004] The steps of the method for detecting and locating small, faint targets in large-size infrared images include:
[0005] S1. Preprocess large-size, low signal-to-noise ratio infrared images by randomly cropping them to reduce the noise in the images and obtain image blocks 1, 2, ..., N that meet the downsampling rate and size requirements.
[0006] S2. Improve the U-net network and design a backbone feature extraction network to obtain a multi-branch feature extraction network. Extract features from the cropped image blocks to obtain feature maps with both high semantic and detailed information.
[0007] S3. Design a centroid localization network, input the feature map, and use the centroid localization network to determine the centroid coordinates.
[0008] S4. The loss function design for the loss value ensures accurate prediction results by using dice loss, which depends on positive samples, in conjunction with focal loss, which depends on negative samples.
[0009] Preferably, step S1 further includes the following sub-steps:
[0010] S101, Image slicing size design: The image slicing size is determined by the target area and the downsampling rate during the feature extraction process;
[0011] S102. Perform random slicing according to the slice size, so that the target appears randomly in any position in the sub-image; when slicing, ensure that the size of the overlapping area is larger than the size of the largest target to prevent the loss of target information;
[0012] S103. Train and infer the image patches to obtain a training set; based on the characteristics of the target, randomly exclude a portion of the slices in the training set that do not contain the target, ensuring that the ratio of positive samples to negative samples is 1:1-2, to prevent the target from being detected in slices that do not contain the target.
[0013] Preferably, step S2 includes:
[0014] S201. Design a U-net network structure with an attention mechanism; the attention mechanism includes channel attention mechanism and spatial attention mechanism;
[0015] The channel attention mechanism performs global max pooling and global average pooling on each channel of the input feature map. Assuming the output feature map size is b*c*w*h, the size after average pooling is b*c*1*1, and the size after max pooling is b*c*1*1. After passing through the fully connected layer, the output size remains unchanged, and the size obtained by adding the output results is b*c*1*1.
[0016] The spatial attention mechanism performs max pooling and average pooling at each position of the input feature map. Assuming the output feature map size is b*c*w*h, the size after average pooling is b*1*w*h, the size after max pooling is b*1*w*h, the size after merging the two in the channel dimension is b*2*w*h, and the size of the output result after passing through the convolutional layer is b*1*w*h.
[0017] S202. Design a multi-branch feature extraction network: Perform channel merging operations on the same layer of different branches to obtain a feature layer that has both general image semantic information and detailed target features; perform channel merging operations on each layer of each branch to obtain a fused feature map based on the U-net network; after completing the above steps, a feature map with the same size as the input image is obtained.
[0018] Preferably, step S3 further includes the following sub-steps:
[0019] S301. Recreate the label based on the centroid coordinate value, set the target centroid position of the two-dimensional all-zero vector to 1, map the semantic segmentation result back to the original image, and obtain a mapped image with the background removed. The mapped image contains only the target pixels and the suspected target pixels.
[0020] S302. Extract image features from the mapped image using two channels with kernel sizes of 3*3 and 5*5 respectively; the channel with kernel size of 5*5 is used to determine the overall features of the target, and the channel with kernel size of 3*3 is used to extract more detailed information from the target.
[0021] S303. The mapped image after convolution is downsampled once to filter out the edge information of the target, and the pixels that contribute more to determining the centroid of the target are obtained. Then, upsampling is performed to amplify the feature points of the target and obtain the feature map.
[0022] S304. Finally, the feature maps at multiple scales are merged to determine the final centroid coordinates.
[0023] Preferably, the loss function in step S4 is:
[0024] LOSS=(dise Loss)+(focal Loss):
[0025]
[0026] In the formula, X represents the pixel matrix of the true value, Y represents the pixel matrix of the predicted result, α is the weight coefficient, and γ is 2.
[0027] Preferably, the downsampling rate in step S101 is 16, and the image size is 512*512.
[0028] The present invention also provides a device for detecting and locating small, faint targets in large infrared images, including a preprocessing module, a feature extraction module, and a centroid localization module;
[0029] Preferably, the preprocessing module is used to perform random image cropping on large-size, low signal-to-noise ratio infrared images; during cropping, the size of the overlapping area must be greater than the size of the largest target to prevent the loss of target information;
[0030] Preferably, the feature extraction module includes a U-net network structure unit with an attention mechanism and a multi-branch feature extraction network unit to obtain feature maps with both high semantic and detailed information.
[0031] Preferably, the centroid localization module is used to accurately locate the centroid of the feature map obtained by the feature extraction module.
[0032] The beneficial effects of this invention are as follows:
[0033] (1) The centroid accuracy is greatly improved, reaching sub-pixel (<1 pixel);
[0034] (2) High robustness;
[0035] (3) It reduced the overall computational load;
[0036] (4) It reduces the complexity of the algorithm. Attached Figure Description
[0037] Figure 1 This is a flowchart illustrating a method for detecting and locating small, faint targets in large infrared images according to an embodiment of the present invention.
[0038] Figure 2 This is a schematic diagram of the feature extraction network for a method of detecting and locating small, dark targets in large infrared images according to an embodiment of the present invention.
[0039] Figure 3 This is a schematic diagram of the centroid localization network for a method for detecting and locating small, faint targets in large infrared images according to an embodiment of the present invention.
[0040] Figure 4 This is a centroid localization network structure diagram of a method for detecting and locating small, faint targets in large infrared images according to an embodiment of the present invention. Detailed Implementation
[0041] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.
[0043] Infrared targets that are faint and indistinct are characterized by their small size, limited texture information, and low signal-to-noise ratio. Currently, the feature extraction networks in commonly used end-to-end target detection frameworks are simply stacks of various convolutional layers. In convolutional neural networks, deep networks readily respond to semantic features but possess limited geometric information, which is detrimental to target localization; shallow networks readily respond to image features but lack semantic features, which is unfavorable for image classification. This invention, by combining the feature extraction capabilities of neural networks, simplifies the "boundary box regression + centroid localization" problem into a direct centroid localization problem, and transforms recognition into a confidence score prediction problem, aiming to achieve high-precision detection and localization of faint targets in large-size, low-signal-to-noise-ratio infrared images end-to-end.
[0044] like Figure 1-4As shown: The purpose of this invention is to provide a method for detecting and locating the centroid of small, faint targets in large-size infrared images. The method includes:
[0045] S1. Preprocess large-size, low signal-to-noise ratio infrared images by randomly cropping them to reduce the noise in the images and obtain image blocks 1, 2, ..., N that meet the downsampling rate and size requirements.
[0046] S2. Improve the U-net network and design a backbone feature extraction network to obtain a multi-branch feature extraction network. Extract features from the cropped image blocks to obtain feature maps with both high semantic and detailed information.
[0047] S3. Design a centroid localization network, input the feature map, and use the centroid localization network to determine the centroid coordinates.
[0048] S4. The loss function design for the loss value ensures accurate prediction results by using dice loss, which depends on positive samples, in conjunction with focal loss, which depends on negative samples.
[0049] Preferably, step S1 further includes the following sub-steps:
[0050] S101, Image slicing size design: The image slicing size is determined by the target area and the downsampling rate during the feature extraction process. Generally, the downsampling rate of feature extraction in target detection frameworks is 32, which means that in the last feature map, one pixel corresponds to a 32*32 area in the original image.
[0051] The downsampling rate in the feature extraction process in step S101 is 16, that is, in the last feature map, one pixel corresponds to a 16*16 area in the original image, and the slice size is 512*512.
[0052] S102. Perform random image slicing according to the image slicing size. By allowing the target to appear randomly in any position in the sub-image, the training diversity is enhanced. When slicing images in step S102, it is necessary to ensure that the size of the overlapping area is larger than the size of the largest target to prevent the loss of target information.
[0053] S103. Training and inference design: Based on the characteristics of the target, a portion of the slices without the target are excluded from the training set to ensure that the ratio of positive to negative samples is 1:1-2, preventing the target from being detected in slices without the target.
[0054] Preferably, each branch of the multi-branch feature extraction network in step S2 is a U-net network structure with an attention mechanism; the attention mechanism is used to enhance the accuracy of target detection, and the feature layers obtained from the branches are fused into a feature map.
[0055] Preferably, the branches use convolutional kernels of different sizes. Different sizes of convolutional kernels can have different receptive fields. The same layer of different branches is merged into a channel merging operation. The merged result is a feature layer that integrates the outputs of multiple convolutional kernels. The result of the feature layer contains both general image semantic information and detailed target features. Each layer is fused from the corresponding layers of all branches to obtain a fused feature map. Finally, a feature map with the same size as the input image is obtained.
[0056] Preferably, in the attention mechanism
[0057] Mc(F) = σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Formula 1
[0058] Ms(F') = σ(f 7x7 (AvgPool(F');MaxPool(F'))) Formula 2
[0059] F” = F' * Ms(F') = (F * Mc(F)) * Ms(F * Mc(F)) Formula 3
[0060] Equation 1 represents the channel attention mechanism, and Equation 2 represents the spatial attention mechanism. Together, they constitute the required attention mechanism. Where F represents the input feature map, F' represents the output feature map, F” represents the final output feature map, σ represents the sigmoid operation, MLP represents fully connected layers, AvgPool represents average pooling, and MaxPool represents max pooling. 7x7 This indicates a convolution operation with a kernel size of 7x7;
[0061] The channel attention mechanism performs global max pooling and global average pooling on each channel of the input feature map. Assuming the output feature map size is b*c*w*h, the size after average pooling is b*c*1*1, and the size after max pooling is b*c*1*1. After passing through the fully connected layer, the output size remains unchanged, and the size obtained by adding the output results is b*c*1*1.
[0062] The spatial attention mechanism performs max pooling and average pooling at each position of the input feature map. Assuming the output feature map size is b*c*w*h, the size after average pooling is b*1*w*h, the size after max pooling is b*1*w*h, the size after merging the two in the channel dimension is b*2*w*h, and the size of the output result after passing through the convolutional layer is b*1*w*h.
[0063] Preferably, the fully convolutional input designed in step S3 has the characteristics of low complexity and low computational cost compared with a typical fully connected layer. Step S3 also includes the following sub-steps:
[0064] S301. Recreate the label based on the centroid coordinate value, set the target centroid position of the 512*512 two-dimensional all-zero vector to 1, and map the semantic segmentation result back to the original image to obtain a background-removed mapping image. The mapping image only contains target pixels and suspected target pixels.
[0065] S302. Extract image features from the mapped image using two channels with convolution kernel sizes of 3*3 and 5*5 respectively. The 5*5 convolution kernel is used to determine the overall features of the target, and the 3*3 convolution kernel is used to extract more detailed information from the target.
[0066] S303. Perform a downsampling operation on the mapped image after convolution to filter out the edge information of the target, obtain the pixels that contribute significantly to determining the centroid of the target, and then perform an upsampling operation to amplify the feature points of the target to obtain the feature map.
[0067] S304. Finally, the feature maps at multiple scales are merged to determine the final centroid coordinates.
[0068] Preferably, the loss function in step S4 uses both die loss and focal loss, and the loss function is:
[0069] LOSS=(dise Loss)+(focal Loss):
[0070]
[0071] Preferably, X represents the pixel matrix of the true values, Y represents the pixel matrix of the predicted results, α is the weighting coefficient, and γ is 2.
[0072] To prevent the network from receiving a small penalty even when all outputs are set to 0, this invention uses dice loss. Dice loss performs well in scenarios with severe imbalance between positive and negative samples and is suitable for semantic segmentation of small objects. For dice loss, when all prediction results are 0... The result is 1, only when the target pixel is predicted correctly. The value will then approach 0. However, when using Dice Loss alone, severe oscillations generally occur when the positive samples are small targets (in the case of only foreground and background, if some pixels are predicted incorrectly, the loss value will fluctuate significantly, resulting in drastic gradient changes). Therefore, this invention uses Dice Loss in conjunction with Focal Loss; for Focal Loss, the loss value is averaged across the entire dataset, relying more on negative samples, thereby improving stability and the realism of the output.
[0073] The present invention also includes a device for detecting and locating small, faint targets in large-size infrared images, comprising a preprocessing module, a feature extraction module, and a centroid localization module;
[0074] The preprocessing module preprocesses large-size, low signal-to-noise ratio infrared images by randomly cropping the images to reduce the noise effect within the images, resulting in several image blocks (sub-images) with a size of 512*512.
[0075] The feature extraction module uses a multi-branch feature extraction network to extract features from the cropped image blocks, obtaining feature maps that have both high semantic and detailed information.
[0076] The feature map of the centroid localization module is input, and the centroid of the target of interest is directly predicted by the centroid localization network and its target confidence is estimated. The confidence score is used to determine whether it is a target.
[0077] The preprocessing module includes:
[0078] Random image slicing enhances training diversity by allowing the target to appear randomly in any position within a sub-image; when slicing, the size of the overlapping region must be greater than the size of the largest target.
[0079] The image slicing size is determined by the target area and the downsampling rate during feature extraction. The downsampling rate during feature extraction is 16, meaning that in the last feature map, one pixel corresponds to a 16*16 area in the original image, and the slice size is 512*512.
[0080] In the training and inference design, during training, because the target area is very small and the distribution is sparse, the training set will exclude some cut images without targets to ensure that the ratio of positive and negative samples is between 1:1 and 1:2, so as to prevent the target from being detected when there are no target cut images.
[0081] The preprocessing unit allows the target to appear randomly at any position in the sub-image, enhancing the random image slicing processing for training diversity; the overlapping area size should be larger than the size of the largest target when slicing; the downsampling rate of the backbone feature extraction network is 16, that is, in the last feature map, one pixel corresponds to a 16*16 area in the original image, and the slice size is set to 512*512.
[0082] The feature extraction module uses a multi-branch feature extraction network where each branch is a U-net network structure with an attention mechanism. Each branch uses a different convolutional kernel size, and different kernel sizes can have different receptive fields. The same layer from different branches is merged, and the merged result is a feature layer that integrates the outputs of multiple convolutional kernels. The result of the feature layer contains both general image semantic information and detailed target features. Each layer is formed by fusing the corresponding layers from all branches to obtain a fused feature map, ultimately resulting in a feature map of the same size as the input image.
[0083] The centroid localization network unit re-labels the target based on the centroid coordinates, setting the centroid position of the 512*512 two-dimensional all-zero vector to 1. This network can greatly reduce the computational cost of centroid calculation. Mapping the semantic segmentation result back to the original image yields an image with the background removed, containing only the target pixels and suspected target pixels. This mapped image is then processed through two channels with convolution kernel sizes of 3*3 and 5*5 to extract image features. The 5*5 convolution kernel determines the overall features of the target, while the 3*3 convolution kernel extracts more detailed information from the target. A downsampling operation is performed on the mapped image after convolution to filter out the target's edge information, obtaining pixels that contribute significantly to determining the target's centroid. This feature map is then upsampled to amplify the target's feature points. Finally, the multi-scale feature maps are merged to determine the final centroid coordinates.
[0084] The loss function unit uses both die loss and focal loss together, with the following loss function:
[0085] LOSS=(dise Loss)+(focal Loss):
[0086]
[0087] Preferably, X represents the pixel matrix of the true values, Y represents the pixel matrix of the predicted results, α is the weighting coefficient, and γ is 2.
[0088] To prevent the network from receiving a small penalty even when all outputs are set to 0, this invention uses dice loss. Dice loss performs well in scenarios with severe imbalance between positive and negative samples and is suitable for semantic segmentation of small objects. For dice loss, when all prediction results are 0... The result is 1, only when the target pixel is predicted correctly. The value will then approach 0. However, when using Dice Loss alone, severe oscillations generally occur when the positive samples are small targets (in the case of only foreground and background, if some pixels are predicted incorrectly, the loss value will fluctuate significantly, resulting in drastic gradient changes). Therefore, this invention uses Dice Loss in conjunction with Focal Loss; for Focal Loss, the loss value is averaged across the entire dataset, relying more on negative samples, thereby improving stability and the realism of the output.
[0089] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
[0090] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for detecting and locating the centroid of small, faint targets in large-size infrared images, characterized in that: The steps of the method include: S1. Preprocess large-size, low signal-to-noise ratio infrared images by randomly cropping them to reduce the noise in the images and obtain image blocks 1, 2, ..., N that meet the downsampling rate and size requirements. S2. Improve the U-net network and design a backbone feature extraction network to obtain a multi-branch feature extraction network. Extract features from the cropped image blocks to obtain feature maps with both high semantic and detailed information. Each branch of the multi-branch feature extraction network in step S2 is a U-net network structure with an attention mechanism; the feature layers obtained from the branches are fused into a feature map; The branches use convolutional kernels of different sizes, and different sizes of convolutional kernels have different receptive fields. The same layer of different branches is merged into a channel merging operation. The merged result is a feature layer that integrates the outputs of multiple convolutional kernels. The result of the feature layer contains both image semantic information and target features. Each layer is fused from the corresponding layers of all branches to obtain a fused feature map. Finally, a feature map with the same size as the input image is obtained. S3. Design a centroid localization network, input the feature map, and use the centroid localization network to determine the centroid coordinates. Step S3 also includes the following sub-steps: S301. Recreate the label based on the centroid coordinate value, set the target centroid position of the two-dimensional all-zero vector to 1, map the semantic segmentation result back to the original image, and obtain a mapped image with the background removed. The mapped image contains only the target pixels and the suspected target pixels. S302, The mapped images are respectively processed by convolution kernels with a size of 3. 3 and 5 Image features are extracted from two channels of a 5x5 convolution kernel; the kernel size is 5. Five channels are used to determine the overall features of the target, and the kernel size is 3. Channel 3 is used to extract more detailed information from the target; S303. The mapped image after convolution is downsampled once to filter out the edge information of the target, and the pixels that contribute more to determining the centroid of the target are obtained. Then, upsampling is performed to amplify the feature points of the target and obtain the feature map. S304. Finally, the feature maps at multiple scales are merged to determine the final centroid coordinates. S4. The loss function design for the loss value ensures accurate prediction results by using dice loss, which depends on positive samples, in conjunction with focal loss, which depends on negative samples.
2. The method for detecting and locating the centroid of small, faint targets in large-size infrared images according to claim 1, characterized in that: Step S1 further includes the following sub-steps: S101, Image slicing size design: The image slicing size is determined by the target area and the downsampling rate during the feature extraction process; S102. Perform random slicing according to the slice size, so that the target appears randomly in any position in the sub-image; when slicing, ensure that the size of the overlapping area is larger than the size of the largest target to prevent the loss of target information; S103. Train and infer the image patches to obtain a training set; based on the characteristics of the target, randomly exclude a portion of the slices in the training set that do not contain the target, ensuring that the ratio of positive samples to negative samples is 1:1-2, to prevent the target from being detected in slices that do not contain the target.
3. The method for detecting and locating the centroid of small, faint targets in large-size infrared images according to claim 2, characterized in that, Step S2 includes: S201. Design a U-net network structure with an attention mechanism; the attention mechanism includes channel attention mechanism and spatial attention mechanism; The channel attention mechanism performs global max pooling and global average pooling on each channel of the input feature map, assuming the output feature map size is b. c w h, the size after average pooling is b c 1 1. The size after max pooling is b c 1 1; The output size remains unchanged after passing through the fully connected layer. The size obtained by adding the output results is b. c 1 1; The spatial attention mechanism performs max pooling and average pooling at each location of the input feature map, assuming the output feature map size is b. c w h, the size after average pooling is b 1 w h, the size after max pooling is b 1 w h, the combined size of the two along the channel dimension is b. 2 w h, the size of the output result after the convolutional layer is b 1 w h; S202. Design a multi-branch feature extraction network: Perform channel merging operations on the same layer of different branches to obtain a feature layer that has both general image semantic information and detailed target features; perform channel merging operations on each layer of each branch to obtain a fused feature map based on the U-net network; after completing the above steps, a feature map with the same size as the input image is obtained.
4. The method for detecting and locating the centroid of small, faint targets in large-size infrared images according to claim 1, characterized in that: The loss function in step S4 is: LOSS=(dise Loss)+(focal Loss): ； In the formula, X represents the pixel matrix of the true value, Y represents the pixel matrix of the predicted result, α is the weight coefficient, and γ is 2.
5. The method for detecting and locating the centroid of small, faint targets in large-size infrared images according to claim 4, characterized in that: In step S101, the downsampling rate is 16 and the slice size is 512. 512.