A method for enhancing the features of small targets in infrared images
Through a single-band backbone network based on saccade and gaze and a multi-scale feature enhancement module, the problem of difficulty in detecting weak targets in infrared images is solved, and the effective enhancement of weak target features in infrared images is achieved, and the detection capability is improved.
Patent Information
- Application Number
- CN202411059751.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-05
AI Technical Summary
The difficulty of detecting weak targets in infrared remote sensing images increases. Due to the weak texture information and low resolution, the information of weak targets is scarce. It is difficult for the prior art to effectively, actively and quickly enhance the characteristics of weak targets in infrared images.
Using a single-band backbone network based on saccade and gaze and a multi-scale feature enhancement module, the infrared image is gradually extracted and enhanced through multiple wavelet transformation modules, splicing modules, convolutional layers and normalized layers. Weak target features in the process, achieving multi-scale correction and fusion of features.
Effectively, proactively and rapidly enhance the characteristics of weak targets in infrared images, improve the detection ability of weak targets in infrared images, and overcome the problems of weak texture information and low resolution in the prior art.
Smart Images

Figure CN118570493B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing, and more particularly to a method for enhancing the features of small targets in infrared images. Background Art
[0002] At present, infrared remote sensing images have the ability to work all day long, but the texture information of infrared remote sensing images is weaker than that of visible light images, which increases the difficulty of detecting weak targets in infrared images. In addition, the resolution of infrared remote sensing images is lower than that of visible light images, resulting in more scarce information on weak targets, which further increases the detection burden of weak targets in infrared images.
[0003] Therefore, how to provide a method for enhancing the features of small targets in infrared images, which can effectively, actively and quickly enhance the features of small targets in infrared images, is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0004] In view of this, an object of the present invention is to provide a method for enhancing features of small targets in infrared images, which can effectively, actively and quickly enhance the features of small targets in infrared images.
[0005] In order to achieve the above object, the present invention adopts the following technical solution:
[0006] A method for enhancing the features of small targets in infrared images comprises the following steps:
[0007] S1: Input the single-band infrared image into the single-band backbone network based on scanning and gaze to obtain the local eye movement areas of each scale and their corresponding features before correction at each scale;
[0008] S2: Inputting the local eye movement areas at each scale and the pre-corrected features at each scale into a multi-scale feature enhancement module based on local eye movement area prompts to obtain enhanced features of weak targets.
[0009] Preferably, the single-band backbone network based on scanning and gaze includes a scanning network and N gaze networks with the same structure, N>=4;
[0010] S1 specifically includes:
[0011] S11: inputting the single-band infrared image into a scanning network to obtain infrared image candidate regions and infrared image features;
[0012] S12: inputting the infrared image candidate region and the infrared image features into a first gaze network to obtain a first-scale local eye movement region and a first-scale pre-correction feature;
[0013] Down-sampling the infrared image candidate region to obtain a first down-sampled infrared image candidate region;
[0014] Inputting the first down-sampled infrared image candidate region and the first scale pre-correction features into a second gaze network to obtain a second-scale local eye movement region and a second-scale pre-correction features;
[0015] Downsampling the first downsampled infrared image candidate region to obtain a second downsampled infrared image candidate region;
[0016] Inputting the second down-sampled infrared image candidate region and the second scale pre-correction features into a third gaze network to obtain a third-scale local eye movement region and a third-scale pre-correction features; ...
[0018] Down-sampling the N-2th down-sampled infrared image candidate region to obtain the N-1th down-sampled infrared image candidate region;
[0019] The N-1th down-sampled infrared image candidate region and the N-1th scale pre-correction features are input into the Nth gaze network to obtain the Nth scale local eye movement region and the Nth scale pre-correction features.
[0020] Preferably, the scanning network includes a plurality of wavelet transform modules, a first splicing module, a first batch of normalization layers, a first SiLU function layer, a candidate region network and a feature extraction module;
[0021] S11 specifically includes:
[0022] Inputting the single-band infrared images into the multiple wavelet transform modules respectively to obtain multiple wavelet transform features;
[0023] After the plurality of wavelet transform features are input into the first splicing module for splicing, they are sequentially passed through the first batch of normalization layers and the first SiLU function layer to obtain infrared image frequency domain features;
[0024] Inputting the infrared image frequency domain features into the candidate region network to obtain the infrared image candidate region;
[0025] The single-band infrared image is input into the feature extraction module to obtain the infrared image features.
[0026] Preferably, the candidate region network includes a first convolution layer, a second SiLu function layer, a second convolution layer, a third SiLu function layer and a Gaussian filter operator; the feature extraction module includes a third convolution layer, a second batch normalization layer and a fourth SiLu function layer;
[0027] S11 also includes:
[0028] After the infrared image frequency domain features are input into the first convolution layer for processing, they are sequentially passed through the second SiLu function layer, the second convolution layer and the third SiLu function layer to obtain a significant area;
[0029] Performing a convolution operation on the salient region and the Gaussian filter operator to obtain the infrared image candidate region;
[0030] After the single-band infrared image is input into the third convolutional layer for processing, it passes through the second batch normalization layer and the fourth SiLu function layer in sequence to obtain the infrared image features.
[0031] Preferably, the first gaze network includes a target local eye movement area prediction module, a global feature extraction module, a local feature extraction module, a second splicing module and a third batch normalization layer;
[0032] S12 specifically includes:
[0033] Inputting the infrared image candidate area into the target local eye movement area prediction module to obtain the first-scale local eye movement area;
[0034] Inputting the infrared image feature into the global feature extraction module to obtain a first global feature;
[0035] After performing pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, the first target-related local feature is obtained through the local feature extraction module;
[0036] Inputting the first target-related local feature and the first global feature into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain a first fusion feature;
[0037] The first fusion feature is added point by point to the infrared image feature to obtain the first feature before scale correction.
[0038] Preferably, the target local eye movement area prediction module includes a fourth convolution layer, a fifth SiLu function layer, a fifth convolution layer and a sixth SiLu function layer; the global feature extraction module includes a sixth convolution layer, a seventh SiLu function layer, a seventh convolution layer and an eighth SiLu function layer; the local feature extraction module includes an eighth convolution layer and a ninth SiLu function layer;
[0039] S12 specifically includes:
[0040] After the infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the first-scale local eye movement area;
[0041] After the infrared image feature is input into the sixth convolution layer for processing, it is sequentially passed through the seventh SiLu function layer, the seventh convolution layer and the eighth SiLu function layer to obtain the first global feature;
[0042] After performing a pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, the first target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer.
[0043] Preferably, the multi-scale feature enhancement module based on local eye movement area prompts includes a feature correction module, a cross-scale response adjustment module and a feature coordination fusion module;
[0044] S2 specifically includes:
[0045] S21: inputting the local eye movement areas at each scale and the pre-correction features at each scale into the feature correction module to obtain the post-correction features at each scale;
[0046] S22: inputting the local eye movement areas of each scale into the cross-scale response adjustment module to obtain the final corresponding weights of each scale;
[0047] S23: Inputting the corrected features of each scale and the final corresponding weights of each scale into the feature coordination fusion module to obtain enhanced features of each scale; the enhanced features of each scale are the enhanced features of the weak target.
[0048] Preferably, the feature correction module includes a local average pooling layer, a standard convolution layer and a variable convolution layer;
[0049] S21 specifically includes:
[0050] Inputting the local eye movement areas of each scale into the local average pooling layer for processing, and then passing through the standard convolution layer to obtain the offset of each scale;
[0051] The scale offsets and the features before scale correction are input into the variable convolution layer to obtain the features after scale correction.
[0052] Preferably, the expression of the cross-scale response adjustment module is:
[0053] ;
[0054] ;
[0055] in, Indicates the final corresponding weight of the i-th scale, Indicates the final corresponding weight of the i-1th scale; represents the initial corresponding weight of the i-th scale; where, ; Indicates the final corresponding weight of the first scale; represents the initial corresponding weight of the first scale; Indicates the input Downsample by a factor of 2 using bilinear interpolation; Indicates that for the input and Take the maximum value of the two corresponding positions.
[0056] Preferably, S23 specifically further includes:
[0057] After point multiplication of the corrected features of each scale and the final corresponding weights of each scale, they are added to the features before correction of each scale to obtain the features before fusion of each scale;
[0058] The pre-fusion features of each scale include N pre-fusion features of different scales; specifically, the pre-fusion features of the first scale, the pre-fusion features of the second scale, ... the pre-fusion features of the Nth scale arranged in order of scale size;
[0059] The scales of the second-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scales of the first-scale pre-fusion features to obtain a first group of pre-fusion features;
[0060] The scales of the first-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scales of the second-scale pre-fusion features to obtain a second set of pre-fusion features;
[0061] The scales of the second-scale pre-fusion features and the fourth-scale pre-fusion features are adjusted to the scale of the third-scale pre-fusion features to obtain a third group of pre-fusion features; ...
[0063] The scales of the N-2th scale pre-fusion features and the Nth scale pre-fusion features are adjusted to the scales of the N-1th scale pre-fusion features to obtain the N-1th group of pre-fusion features;
[0064] Adjust the scales of the N-1th scale pre-fusion features and the N-2th scale pre-fusion features to the scale of the Nth scale pre-fusion features; obtain the Nth group of pre-fusion features;
[0065] Each set of pre-fusion features is input into the standard convolution layer for processing, and then passed through the LeaklyReLU activation function layer to obtain the enhanced features at each scale.
[0066] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for enhancing the features of small targets in infrared images, which can effectively, actively and quickly enhance the features of small targets in infrared images. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0068] Figure 1 A schematic diagram of a method for enhancing the features of a small target in an infrared image provided in an embodiment of the present invention;
[0069] Figure 2 A schematic diagram of a single-band backbone network based on scanning and gaze provided in an embodiment of the present invention;
[0070] Figure 3 A schematic diagram of a scanning network provided in an embodiment of the present invention;
[0071] Figure 4 A schematic diagram of a candidate area network provided in an embodiment of the present invention;
[0072] Figure 5 A schematic diagram of a feature extraction module provided in an embodiment of the present invention;
[0073] Figure 6 A schematic diagram of a gaze network provided in an embodiment of the present invention;
[0074] Figure 7 A schematic diagram of a target local eye movement area prediction module provided in an embodiment of the present invention;
[0075] Figure 8 A schematic diagram of a global feature extraction module provided in an embodiment of the present invention;
[0076] Fig. 9 A schematic diagram of a local feature extraction module provided in an embodiment of the present invention;
[0077] Fig.10 A schematic diagram of a multi-scale feature enhancement module based on local eye movement area prompts provided in an embodiment of the present invention;
[0078] Fig.11A schematic diagram of a feature correction module provided in an embodiment of the present invention;
[0079] Fig.12 A schematic diagram of obtaining each group of pre-fusion features provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] like Figure 1 As shown, the embodiment of the present invention discloses a method for enhancing the features of small targets in infrared images, comprising the following steps:
[0082] S1: Input the single-band infrared image into the single-band backbone network based on scanning and gaze to obtain the local eye movement areas of each scale and their corresponding features before correction at each scale;
[0083] S2: Inputting the local eye movement areas at each scale and the pre-corrected features at each scale into a multi-scale feature enhancement module based on local eye movement area prompts to obtain enhanced features of weak targets.
[0084] It can be understood that a single-band infrared image I does not contain color information and contains only one band, and its dimension satisfies , where R is a real number, H, W and 1 are the height, width and dimension of the single-band infrared image I respectively;
[0085] Single-band infrared images include long-wave infrared images, medium-wave infrared images, and simulated infrared images generated by simulation technology. Among them, the generation of simulated infrared images can be based on data from the visible light band, or combined with infrared images and visible light images.
[0086] In one embodiment, if Figure 2 As shown, the single-band backbone network based on scanning and gaze includes a scanning network and N gaze networks with the same structure, N>=4;
[0087] S1 specifically includes:
[0088] S11: inputting the single-band infrared image into a scanning network to obtain infrared image candidate regions and infrared image features;
[0089] S12: inputting the infrared image candidate region and the infrared image features into a first gaze network to obtain a first-scale local eye movement region and a first-scale pre-correction feature;
[0090] Down-sampling the infrared image candidate region to obtain a first down-sampled infrared image candidate region;
[0091] Inputting the first down-sampled infrared image candidate region and the first scale pre-correction features into a second gaze network to obtain a second-scale local eye movement region and a second-scale pre-correction features;
[0092] Downsampling the first downsampled infrared image candidate region to obtain a second downsampled infrared image candidate region;
[0093] Inputting the second down-sampled infrared image candidate region and the second scale pre-correction features into a third gaze network to obtain a third-scale local eye movement region and a third-scale pre-correction features; ...
[0095] Down-sampling the N-2th down-sampled infrared image candidate region to obtain the N-1th down-sampled infrared image candidate region;
[0096] The N-1th down-sampled infrared image candidate region and the N-1th scale pre-correction features are input into the Nth gaze network to obtain the Nth scale local eye movement region and the Nth scale pre-correction features.
[0097] In one embodiment, if Figure 3 As shown, the scanning network includes multiple wavelet transform modules, a first splicing module, a first batch of normalization layers, a first SiLU function layer, a candidate region network and a feature extraction module;
[0098] S11 specifically includes:
[0099] Inputting the single-band infrared images into the multiple wavelet transform modules respectively to obtain multiple wavelet transform features;
[0100] After the plurality of wavelet transform features are input into the first splicing module for splicing, they are sequentially passed through the first batch of normalization layers and the first SiLU function layer to obtain infrared image frequency domain features;
[0101] Inputting the infrared image frequency domain features into the candidate region network to obtain the infrared image candidate region;
[0102] The single-band infrared image is input into the feature extraction module to obtain the infrared image features.
[0103] In this embodiment: the first SiLU function layer adopts a nonlinear transformation SiLU function.
[0104] In this embodiment, the calculation formula of the feature F obtained by combining multiple wavelet transform features is:
[0105] ;
[0106] Where I represents a single-band infrared image, d represents the number of wavelet transform types included in the scanning network (i.e., the number of wavelet transform modules included in the scanning network. In the present invention, one wavelet transform module corresponds to one type of wavelet transform), and c i represents the number of parameters used by the i-th wavelet transform type, ∪ represents the dimension concatenation operation, T represents the matrix transposition transformation, and * represents the convolution operation. and They represent the row transformation parameter and column transformation parameter of the jth parameter used by the i-th wavelet transform type respectively; and They represent the high-frequency and low-frequency features extracted from a single-band infrared image by row-wise decomposition, and They respectively represent the high-frequency features and low-frequency features extracted from the single-band infrared image by decomposition in the column direction. Whether it is row decomposition or column decomposition, it will cause the width and height of the single-band infrared image to be reduced to half of the original.
[0107] In this embodiment, the present invention selects four different types of wavelet transforms, namely Harr transform, DB2 transform, Coiflet transform and BiorNr.Nd transform. Each wavelet transform produces four different feature decomposition results, which reflect the high-frequency characteristics and low-frequency characteristics of the image, specifically including: the positive component of the high frequency in the row direction and the low frequency in the column direction, the negative component of the high frequency in the row direction and the low frequency in the column direction, the positive component of the low frequency in the row direction and the high frequency in the column direction, and the negative component of the low frequency in the row direction and the high frequency in the column direction.
[0108] The c corresponding to the above four wavelet transforms i The value of The initial value of The initial values are shown in Table 1:
[0109] Table 1 c corresponding to four wavelet transforms i The value of The initial value of Initial value of
[0110]
[0111] In one embodiment, if Figure 4 and Figure 5As shown, the candidate region network includes a first convolution layer, a second SiLu function layer, a second convolution layer, a third SiLu function layer and a Gaussian filter operator; the feature extraction module includes a third convolution layer, a second batch normalization layer and a fourth SiLu function layer;
[0112] S11 also includes:
[0113] After the infrared image frequency domain features are input into the first convolution layer for processing, they are sequentially passed through the second SiLu function layer, the second convolution layer and the third SiLu function layer to obtain a significant area;
[0114] Performing a convolution operation on the salient region and the Gaussian filter operator to obtain the infrared image candidate region;
[0115] After the single-band infrared image is input into the third convolutional layer for processing, it passes through the second batch normalization layer and the fourth SiLu function layer in sequence to obtain the infrared image features.
[0116] In this embodiment: the convolution kernel size of the first convolution layer is 3*3; the convolution kernel size of the second convolution layer is 1*1; the convolution kernel size of the third convolution layer is 3*3; the second SiLU function layer, the third SiLU function layer and the fourth SiLU function layer all use nonlinear transformation SiLU function.
[0117] In one embodiment, if Figure 6 As shown, the first gaze network includes a target local eye movement area prediction module, a global feature extraction module, a local feature extraction module, a second splicing module and a third batch normalization layer;
[0118] S12 specifically includes:
[0119] Inputting the infrared image candidate area into the target local eye movement area prediction module to obtain the first-scale local eye movement area;
[0120] Inputting the infrared image feature into the global feature extraction module to obtain a first global feature;
[0121] After performing pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, the first target-related local feature is obtained through the local feature extraction module;
[0122] Inputting the first target-related local feature and the first global feature into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain a first fusion feature;
[0123] The first fusion feature is added point by point to the infrared image feature to obtain the first feature before scale correction.
[0124] It can be understood that: because the second gaze network, the third gaze network...the Nth gaze network all have the same structure as the first gaze network;
[0125] Therefore, S12 also specifically includes:
[0126] Inputting the first down-sampled infrared image candidate area into the target local eye movement area prediction module to obtain the second-scale local eye movement area;
[0127] Inputting the first pre-scale correction feature into the global feature extraction module to obtain a second global feature;
[0128] After performing a pixel-by-pixel dot multiplication operation on the second-scale local eye movement area and the second global feature, the second target-related local feature is obtained by the local feature extraction module;
[0129] Inputting the second target-related local features and the second global features into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain second fusion features;
[0130] The second fused feature is added point by point to the first pre-scale correction feature to obtain the second pre-scale correction feature.
[0131] Inputting the second down-sampled infrared image candidate area into the target local eye movement area prediction module to obtain the third-scale local eye movement area;
[0132] Inputting the second pre-scale correction feature into the global feature extraction module to obtain a third global feature;
[0133] After performing pixel-by-pixel dot multiplication operation on the third-scale local eye movement area and the third global feature, a third target-related local feature is obtained through the local feature extraction module;
[0134] Inputting the third target-related local feature and the third global feature into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain a third fusion feature;
[0135] The third fused feature is added point by point to the second feature before scale correction to obtain the third feature before scale correction. ...
[0137] Inputting the N-1th down-sampled infrared image candidate area into the target local eye movement area prediction module to obtain the Nth scale local eye movement area;
[0138] Inputting the N-1th pre-scale correction feature into the global feature extraction module to obtain the Nth global feature;
[0139] After performing pixel-by-pixel dot multiplication operation on the Nth scale local eye movement area and the Nth global feature, the Nth target-related local feature is obtained through the local feature extraction module;
[0140] Inputting the Nth target-related local feature and the Nth global feature into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain the Nth fused feature;
[0141] The Nth fused feature and the N-1th pre-scale correction feature are added point by point to obtain the Nth pre-scale correction feature.
[0142] In one embodiment, if Figure 7-Figure 9 As shown, the target local eye movement area prediction module includes a fourth convolution layer, a fifth SiLu function layer, a fifth convolution layer and a sixth SiLu function layer; the global feature extraction module includes a sixth convolution layer, a seventh SiLu function layer, a seventh convolution layer and an eighth SiLu function layer; the local feature extraction module includes an eighth convolution layer and a ninth SiLu function layer;
[0143] S12 specifically includes:
[0144] After the infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the first-scale local eye movement area;
[0145] After the infrared image feature is input into the sixth convolution layer for processing, it is sequentially passed through the seventh SiLu function layer, the seventh convolution layer and the eighth SiLu function layer to obtain the first global feature;
[0146] After performing a pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, the first target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer.
[0147] It can be understood that: because the second gaze network, the third gaze network...the Nth gaze network all have the same structure as the first gaze network;
[0148] Therefore, the target local eye movement area prediction module structure in the second gaze network, the third gaze network...the Nth gaze network is exactly the same as the target local eye movement area prediction module structure in the first gaze network;
[0149] Therefore, S12 also specifically includes:
[0150] After the first down-sampled infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the second-scale local eye movement area;
[0151] Inputting the first pre-scale correction feature into the sixth convolutional layer for processing, and then sequentially passing through the seventh SiLu function layer, the seventh convolutional layer and the eighth SiLu function layer to obtain the second global feature;
[0152] After performing a pixel-by-pixel dot multiplication operation on the second-scale local eye movement area and the second global feature, the second-target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer.
[0153] After the second down-sampled infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the third-scale local eye movement area;
[0154] Inputting the second pre-scale correction feature into the sixth convolutional layer for processing, and then sequentially passing through the seventh SiLu function layer, the seventh convolutional layer and the eighth SiLu function layer to obtain the third global feature;
[0155] After performing a pixel-by-pixel dot multiplication operation on the third-scale local eye movement area and the third global feature, the third target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer. ...
[0157] After the N-1th down-sampled infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the Nth scale local eye movement area;
[0158] Inputting the N-1th pre-scale correction feature into the sixth convolutional layer for processing, and then sequentially passing through the seventh SiLu function layer, the seventh convolutional layer and the eighth SiLu function layer to obtain the Nth global feature;
[0159] After performing pixel-by-pixel dot multiplication operation on the Nth scale local eye movement area and the Nth global feature, the Nth target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer.
[0160] In this embodiment: the convolution kernel sizes of the fourth convolution layer, the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, and the eighth convolution layer are all 3*3;
[0161] The fifth SiLu function layer, the sixth SiLu function layer, the seventh SiLu function layer, the eighth SiLu function layer and the ninth SiLu function layer all adopt nonlinear transformation SiLU functions.
[0162] In one embodiment, if Fig.10 As shown, the multi-scale feature enhancement module based on local eye movement area prompts includes a feature correction module, a cross-scale response adjustment module and a feature coordination fusion module;
[0163] S2 specifically includes:
[0164] S21: inputting the local eye movement areas at each scale and the pre-correction features at each scale into the feature correction module to obtain the post-correction features at each scale;
[0165] S22: inputting the local eye movement areas of each scale into the cross-scale response adjustment module to obtain the final corresponding weights of each scale;
[0166] S23: Inputting the corrected features of each scale and the final corresponding weights of each scale into the feature coordination fusion module to obtain enhanced features of each scale; the enhanced features of each scale are the enhanced features of the weak target.
[0167] In one embodiment, if Fig.11 As shown, the feature correction module includes a local average pooling layer, a standard convolution layer and a variable convolution layer;
[0168] S21 specifically includes:
[0169] Inputting the local eye movement areas of each scale into the local average pooling layer for processing, and then passing through the standard convolution layer to obtain the offset of each scale;
[0170] The scale offsets and the features before scale correction are input into the variable convolution layer to obtain the features after scale correction.
[0171] In one embodiment, the expression of the cross-scale response adjustment module is:
[0172] ;
[0173] ;
[0174] in, Indicates the final corresponding weight of the i-th scale, Indicates the final corresponding weight of the i-1th scale; represents the initial corresponding weight of the i-th scale; where, ; Indicates the final corresponding weight of the first scale; represents the initial corresponding weight of the first scale; Indicates the input Downsample by a factor of 2 using bilinear interpolation; Indicates that for the input and Take the maximum value of the two corresponding positions.
[0175] It can be understood that: the cross-scale response adjustment module is used to adjust the initial weight corresponding to each scale feature, so as to update the corresponding area that needs extra attention for each scale feature; the initial weight corresponding to each size feature is initialized by its corresponding local eye movement area, and the multi-scale initial corresponding weights are arranged from large to small according to the weight size. It can be seen from the adjustment expression that the final corresponding weight of each scale is calculated based on its own initial weight and all initial weights of scales larger than itself;
[0176] In one embodiment, S23 specifically further includes:
[0177] After point multiplication of the corrected features of each scale and the final corresponding weights of each scale, they are added to the features before correction of each scale to obtain the features before fusion of each scale;
[0178] The pre-fusion features of each scale include N pre-fusion features of different scales; specifically, the pre-fusion features of the first scale, the pre-fusion features of the second scale, ... the pre-fusion features of the Nth scale arranged in order of scale size;
[0179] like Fig.12 As shown, the scales of the second-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scales of the first-scale pre-fusion features (implemented by the nearest neighbor algorithm) to obtain a first set of pre-fusion features;
[0180] It can be understood that: the adjusted second-scale pre-fusion features, the adjusted third-scale pre-fusion features and the first-scale pre-fusion features, the three pre-fusion features of the same scale constitute the first group of pre-fusion features;
[0181] The scales of the first-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scale of the second-scale pre-fusion features (implemented by the nearest neighbor algorithm), so as to obtain a second set of pre-fusion features;
[0182] It can be understood that: the adjusted first-scale pre-fusion features, the adjusted third-scale pre-fusion features and the second-scale pre-fusion features, the three pre-fusion features of the same scale constitute the second group of pre-fusion features;
[0183] The scales of the second-scale pre-fusion features and the fourth-scale pre-fusion features are adjusted to the scale of the third-scale pre-fusion features (implemented by the nearest neighbor algorithm), so as to obtain a third set of pre-fusion features;
[0184] It can be understood that: the adjusted second-scale pre-fusion features, the adjusted fourth-scale pre-fusion features and the third-scale pre-fusion features, the three pre-fusion features of the same scale constitute the third group of pre-fusion features; ...
[0186] The scales of the N-2th scale pre-fusion features and the Nth scale pre-fusion features are adjusted to the scale of the N-1th scale pre-fusion features (implemented by the nearest neighbor algorithm), and the N-1th group of pre-fusion features is obtained;
[0187] It can be understood that: the adjusted pre-fusion features of the N-2th scale, the adjusted pre-fusion features of the Nth scale and the pre-fusion features of the N-1th scale, the pre-fusion features of the three scales, constitute the N-1th group of pre-fusion features;
[0188] Adjust the scales of the N-1th scale pre-fusion features and the N-2th scale pre-fusion features to the scale of the Nth scale pre-fusion features (implemented by the nearest neighbor algorithm); obtain the Nth group of pre-fusion features;
[0189] It can be understood that: the adjusted pre-fusion features of the N-1th scale, the adjusted pre-fusion features of the N-2th scale and the pre-fusion features of the Nth scale, the three pre-fusion features of the same scale constitute the Nth group of pre-fusion features;
[0190] Each set of pre-fusion features is input into the standard convolution layer for processing, and then passed through the LeaklyReLU activation function layer to obtain the enhanced features at each scale.
[0191] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0192] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for enhancing the features of small targets in infrared images, characterized in that: The following steps are involved: S1: Input the single-band infrared image into the single-band backbone network based on scanning and gaze to obtain the local eye movement areas of each scale and their corresponding features before correction at each scale; The single-band backbone network based on scanning and gaze includes a scanning network and N gaze networks with the same structure, where N>=4; S1 specifically includes: S11: inputting the single-band infrared image into a scanning network to obtain infrared image candidate regions and infrared image features; S12: inputting the infrared image candidate region and the infrared image features into a first gaze network to obtain a first-scale local eye movement region and a first-scale pre-correction feature; Down-sampling the infrared image candidate region to obtain a first down-sampled infrared image candidate region; Inputting the first down-sampled infrared image candidate region and the first scale pre-correction features into a second gaze network to obtain a second-scale local eye movement region and a second-scale pre-correction features; Downsampling the first downsampled infrared image candidate region to obtain a second downsampled infrared image candidate region; Inputting the second down-sampled infrared image candidate region and the second scale pre-correction features into a third gaze network to obtain a third-scale local eye movement region and a third-scale pre-correction features; Down-sampling the N-2th down-sampled infrared image candidate region to obtain the N-1th down-sampled infrared image candidate region; Inputting the N-1th down-sampled infrared image candidate region and the N-1th scale pre-correction features into the Nth gaze network to obtain the Nth scale local eye movement region and the Nth scale pre-correction features; The scanning network includes multiple wavelet transform modules, a first splicing module, a first batch of normalization layers, a first SiLU function layer, a candidate region network and a feature extraction module; S11 specifically includes: Inputting the single-band infrared images into the multiple wavelet transform modules respectively to obtain multiple wavelet transform features; After the plurality of wavelet transform features are input into the first splicing module for splicing, they are sequentially passed through the first batch of normalization layers and the first SiLU function layer to obtain infrared image frequency domain features; Inputting the infrared image frequency domain features into the candidate region network to obtain the infrared image candidate region; Inputting the single-band infrared image into the feature extraction module to obtain the infrared image features; S2: Inputting the local eye movement areas at each scale and the pre-corrected features at each scale into a multi-scale feature enhancement module based on local eye movement area prompts to obtain enhanced features of weak targets; The multi-scale feature enhancement module based on local eye movement area prompts includes a feature correction module, a cross-scale response adjustment module and a feature coordination fusion module; S2 specifically includes: S21: inputting the local eye movement areas at each scale and the pre-correction features at each scale into the feature correction module to obtain the corrected features at each scale; The feature correction module includes a local average pooling layer, a standard convolution layer and a variable convolution layer; S21 specifically includes: Inputting the local eye movement areas of each scale into the local average pooling layer for processing, and then passing through the standard convolution layer to obtain the offset of each scale; Inputting the offsets of each scale and the features before correction of each scale into the variable convolution layer accordingly to obtain the features after correction of each scale; S22: Inputting the local eye movement areas of each scale into the cross-scale response adjustment module to obtain the final corresponding weights of each scale; The expression of the cross-scale response adjustment module is: in, Indicates the final corresponding weight of the i-th scale, represents the final corresponding weight of the i-1th scale; W i represents the initial corresponding weight of the i-th scale; where 2≤i≤N; represents the final corresponding weight of the first scale; W1 represents the initial corresponding weight of the first scale; Indicates the input Downsample by a factor of 2 using bilinear interpolation; Indicates that for the input and W i Take the maximum value of the corresponding positions of the two; S23: Inputting the corrected features of each scale and the final corresponding weights of each scale into the feature coordination fusion module to obtain enhanced features of each scale; the enhanced features of each scale are the enhanced features of the weak target; S23 specifically includes: After point multiplication of the corrected features of each scale and the final corresponding weights of each scale, they are added to the features before correction of each scale to obtain the features before fusion of each scale; The pre-fusion features of each scale include N pre-fusion features of different scales; specifically, the pre-fusion features of the first scale, the pre-fusion features of the second scale, and the pre-fusion features of the Nth scale arranged in order of scale size; The scales of the second-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scales of the first-scale pre-fusion features to obtain a first group of pre-fusion features; The scales of the first-scale pre-fusion features and the third-scale pre-fusion features are adjusted to the scales of the second-scale pre-fusion features to obtain a second set of pre-fusion features; The scales of the second-scale pre-fusion features and the fourth-scale pre-fusion features are adjusted to the scale of the third-scale pre-fusion features to obtain a third group of pre-fusion features; The scales of the N-2th scale pre-fusion features and the Nth scale pre-fusion features are adjusted to the scales of the N-1th scale pre-fusion features to obtain the N-1th group of pre-fusion features; Adjust the scales of the N-1th scale pre-fusion features and the N-2th scale pre-fusion features to the scale of the Nth scale pre-fusion features; obtain the Nth group of pre-fusion features; Each set of pre-fusion features is input into the standard convolution layer for processing, and then passed through the LeaklyReLU activation function layer to obtain the enhanced features at each scale.
2. The method for enhancing the features of small targets in infrared images according to claim 1, characterized in that: The candidate region network includes a first convolution layer, a second SiLu function layer, a second convolution layer, a third SiLu function layer and a Gaussian filter operator; the feature extraction module includes a third convolution layer, a second batch normalization layer and a fourth SiLu function layer; S11 also includes: After the infrared image frequency domain features are input into the first convolution layer for processing, they are sequentially passed through the second SiLu function layer, the second convolution layer and the third SiLu function layer to obtain a significant area; Performing a convolution operation on the salient region and the Gaussian filter operator to obtain the infrared image candidate region; After the single-band infrared image is input into the third convolutional layer for processing, it passes through the second batch normalization layer and the fourth SiLu function layer in sequence to obtain the infrared image features.
3. The method for enhancing the features of small targets in infrared images according to claim 1, characterized in that: The first gaze network includes a target local eye movement area prediction module, a global feature extraction module, a local feature extraction module, a second splicing module and a third batch normalization layer; S12 specifically includes: Inputting the infrared image candidate area into the target local eye movement area prediction module to obtain the first-scale local eye movement area; Inputting the infrared image feature into the global feature extraction module to obtain a first global feature; After performing pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, a first target-related local feature is obtained through the local feature extraction module; Inputting the first target-related local feature and the first global feature into the second stitching module for feature stitching, and then passing through the third batch normalization layer to obtain a first fusion feature; The first fusion feature is added point by point to the infrared image feature to obtain the first feature before scale correction.
4. The method for enhancing the features of small targets in infrared images according to claim 3, characterized in that: The target local eye movement area prediction module includes a fourth convolution layer, a fifth SiLu function layer, a fifth convolution layer and a sixth SiLu function layer; the global feature extraction module includes a sixth convolution layer, a seventh SiLu function layer, a seventh convolution layer and an eighth SiLu function layer; the local feature extraction module includes an eighth convolution layer and a ninth SiLu function layer; S12 specifically includes: After the infrared image candidate area is input into the fourth convolution layer for processing, it is sequentially passed through the fifth SiLu function layer, the fifth convolution layer and the sixth SiLu function layer to obtain the first-scale local eye movement area; After the infrared image feature is input into the sixth convolution layer for processing, it is sequentially passed through the seventh SiLu function layer, the seventh convolution layer and the eighth SiLu function layer to obtain the first global feature; After performing a pixel-by-pixel dot multiplication operation on the first-scale local eye movement area and the first global feature, the first target-related local feature is obtained by sequentially passing through the eighth convolutional layer and the ninth SiLu function layer.
