Weak target detection method and device based on infrared medium-long wave fusion and medium

Abstract

The embodiment of the application discloses a weak and small target detection method and device based on infrared medium-long wave fusion and a medium; the method can comprise: carrying out discrete wavelet change decomposition on an original medium-long wave infrared image to obtain high-frequency data and low-frequency data; carrying out maximum absolute value fusion on the high-frequency data to obtain a significant part; carrying out weighted average fusion on the low-frequency data to obtain a contour part; fusing the significant part with an L matrix obtained through LatLRR training to obtain detailed information; modulating the significant part and the detailed information based on gray contrast to obtain a significant feature; fusing the significant feature and the contour part to obtain a reconstructed image; and carrying out target recognition on the reconstructed image based on a U-Transformer model combined by a U-shaped neural network and a Transformer algorithm to obtain position information of a weak and small target.

Description

Technical Field

[0001] The present invention relates to the field of infrared image processing technology, and in particular to a method, device and medium for detecting weak targets based on infrared mid- and long-wave fusion. Background Technology

[0002] With the development of image fusion, fusion algorithms have evolved from traditional manual design to the current intelligent deep network training, significantly improving the quality of fused images. Fusion algorithms based on human experience have strong mathematical interpretability and comprehensive failure and debugging analysis, but they cannot cope with complex and changing environments and have poor robustness. Deep learning methods applied to image fusion mainly utilize deep convolutional networks to train on datasets, ultimately extracting image detail information and fusing the images.

[0003] While deep learning methods effectively address the issues of insufficient image detail and blurred edges in infrared images, and eliminate the influence of subjective human factors, they are not particularly effective at highlighting the characteristics of weak targets and suppressing background information. Furthermore, deep learning algorithms rely on a large amount of labeled data for training.

[0004] In the field of infrared weak target detection, existing traditional algorithms and semantic segmentation-based algorithms do not pay much attention to post-processing. These algorithms pursue high detection rates, leading to high false alarm rates, and post-processing requires human prior knowledge. Semantic segmentation-based algorithms cannot achieve end-to-end data output, and traditional algorithms are slow in processing. Post-processing requires pixel-by-pixel calculations on the infrared image, all of which result in poor real-time performance and failure to meet relevant technical requirements. Summary of the Invention

[0005] In view of this, the embodiments of the present invention aim to provide a method, device and medium for detecting weak targets based on infrared mid- and long-wave fusion; it can reduce the involvement of human experience and improve robustness; and achieve end-to-end data output. The real-time performance of the algorithm can also be guaranteed through hardware acceleration, and the algorithm's ability to detect weak infrared targets in complex scenarios is improved.

[0006] The technical solution of this invention is implemented as follows:

[0007] In a first aspect, embodiments of the present invention provide a method for detecting weak targets based on infrared mid-to-long-wave fusion, the method comprising:

[0008] The original mid- and long-wave infrared images were decomposed using discrete wavelet transform to obtain high-frequency and low-frequency data respectively.

[0009] The significant portion is obtained by fusing the high-frequency data using the maximum absolute value;

[0010] The contour portion is obtained by weighted averaging and fusing the low-frequency data.

[0011] The salient part is fused with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0012] Based on grayscale contrast modulation, salient features are obtained by combining the salient parts and the detailed information.

[0013] The salient features are fused with the contour portion to obtain a reconstructed image;

[0014] Based on the combination of U-shaped neural network and Transformer algorithm, the original mid- and long-wave infrared images are decomposed by discrete wavelet transform to obtain high-frequency data and low-frequency data respectively.

[0015] The significant portion is obtained by fusing the high-frequency data using the maximum absolute value;

[0016] The contour portion is obtained by weighted averaging and fusing the low-frequency data.

[0017] The salient part is fused with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0018] Based on grayscale contrast modulation, salient features are obtained by combining the salient parts and the detailed information.

[0019] The salient features are fused with the contour portion to obtain a reconstructed image;

[0020] The U-Transformer model, which combines a U-shaped neural network with the Transformer algorithm, is used to perform target recognition on the reconstructed image and obtain the location information of weak targets.

[0021] Secondly, embodiments of the present invention provide a weak target detection device based on infrared mid-to-long wave fusion, the device comprising: a wavelet transform decomposition part, a first fusion part, a second fusion part, a third fusion part, a modulation part, a fourth fusion part, and a recognition part; wherein,

[0022] The wavelet transform decomposition part is configured to perform discrete wavelet transform decomposition on the original mid- and long-wave infrared image to obtain high-frequency data and low-frequency data respectively.

[0023] The first fusion portion is configured to perform maximum absolute value fusion on the high-frequency data to obtain a significant portion;

[0024] The second fusion part is configured to perform weighted average fusion of the low-frequency data to obtain the contour part;

[0025] The third fusion part is configured to fuse the salient part with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0026] The modulation portion is configured to obtain salient features based on grayscale contrast modulation according to the salient portion and the detailed information;

[0027] The fourth fusion part is configured to fuse the salient features with the contour part to obtain a reconstructed image;

[0028] The identification part is configured as a wavelet transform decomposition part based on a combination of a U-shaped neural network and the Transformer algorithm, a first fusion part, a second fusion part, a third fusion part, a modulation part, a fourth fusion part, and an identification part; wherein,

[0029] The wavelet transform decomposition part is configured to perform discrete wavelet transform decomposition on the original mid- and long-wave infrared image to obtain high-frequency data and low-frequency data respectively.

[0030] The first fusion portion is configured to perform maximum absolute value fusion on the high-frequency data to obtain a significant portion;

[0031] The second fusion part is configured to perform weighted average fusion of the low-frequency data to obtain the contour part;

[0032] The third fusion part is configured to fuse the salient part with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0033] The modulation portion is configured to obtain salient features based on grayscale contrast modulation according to the salient portion and the detailed information;

[0034] The fourth fusion part is configured to fuse the salient features with the contour part to obtain a reconstructed image;

[0035] The recognition part is configured to perform target recognition on the reconstructed image based on a U-Transformer model that combines a U-shaped neural network and a Transformer algorithm, and obtain the location information of weak targets.

[0036] Thirdly, embodiments of the present invention provide a computing device, including: a communication interface, a memory, and a processor; the various components are coupled together via a bus system; wherein,

[0037] The communication interface is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0038] The memory is used to store computer programs that can run on the processor;

[0039] The processor is used to execute the steps of the weak target detection method based on infrared mid- and long-wave fusion described in the above technical solution when running the computer program.

[0040] Fourthly, embodiments of the present invention provide a computer storage medium storing a weak target detection program based on infrared mid- and long-wave fusion. When the weak target detection program based on infrared mid- and long-wave fusion is executed by at least one processor, it implements the steps of the weak target detection method based on infrared mid- and long-wave fusion described in the first aspect.

[0041] This invention provides a method, device, and medium for detecting weak targets based on infrared mid-to-long-wave fusion. By classifying infrared images into mid-to-long-wave grades and reconstructing salient and contour parts, it effectively improves the problem that infrared weak target fusion algorithms cannot suppress background-highlighted targets. Secondly, by using LatLRR training, it solves the problem of weak robustness caused by human experience involvement, and at the same time has stronger mathematical interpretability compared to deep learning algorithms. Finally, based on the prior information that infrared weak targets exhibit a Gaussian distribution, a U-Transformer model is proposed, which effectively improves the algorithm's ability to detect infrared weak targets in complex scenes. Attached Figure Description

[0042] Figure 1 A schematic diagram of a weak target detection method based on infrared mid- and long-wave fusion provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram illustrating the image fusion stage processing provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the composition of the U-Transformer model provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the Transformer component provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram illustrating the composition of the W-TF module or SW-TF module provided in the embodiments of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of a multilayer sensor provided in an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of the detection results of a simulation example provided in an embodiment of the present invention;

[0049] Figure 8 A schematic diagram of a weak target detection device based on infrared mid- and long-wave fusion provided in an embodiment of the present invention;

[0050] Figure 9 A schematic diagram of another weak target detection device based on infrared mid- and long-wave fusion provided in an embodiment of the present invention;

[0051] Figure 10 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0053] See Figure 1 This illustrates a method for detecting weak targets based on infrared mid- and long-wave fusion provided by an embodiment of the present invention. The method includes:

[0054] S101: The original mid- and long-wave infrared image is decomposed by discrete wavelet transform to obtain high-frequency data and low-frequency data respectively;

[0055] S102: The high-frequency data is fused using the maximum absolute value to obtain the significant portion;

[0056] S103: The low-frequency data is weighted and averaged to obtain the contour portion;

[0057] S104: Fuse the salient part with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0058] S105: Obtain salient features based on grayscale contrast modulation according to the salient parts and the detailed information;

[0059] S106: Fuse the salient features with the contour portion to obtain a reconstructed image;

[0060] S107: The U-Transformer model, which combines a U-shaped neural network and the Transformer algorithm, is used to perform target recognition on the reconstructed image to obtain the location information of weak targets.

[0061] The above scheme effectively addresses the problem that infrared weak target fusion algorithms cannot suppress background prominence by classifying infrared images into mid- and long-wavelength categories and reconstructing salient and contour parts. Secondly, LatLRR training solves the problem of weak robustness caused by human experience involvement and has stronger mathematical interpretability compared to deep learning algorithms. Finally, based on the prior information that infrared weak targets exhibit a Gaussian distribution, a U-Transformer model is proposed, which effectively improves the algorithm's ability to detect infrared weak targets in complex scenes.

[0062] It should be noted that, Figure 1 The technical solution shown can be divided into two stages: an image fusion stage and a target recognition stage; wherein, S101 to S105 disclose the technical features included in the image fusion stage; and S106 discloses the technical features of the target recognition stage.

[0063] For the image fusion stage, the technical solution of this invention employs Discrete Wavelet Transform (DWT) and Latent Low-Rank Representation (LatLRR) for image fusion. Specifically, after acquiring the original infrared image, the original infrared image is divided into an original mid-wave infrared image and an original long-wave infrared image; DWT processing is performed on the original mid-wave infrared image and the original long-wave infrared image respectively, thereby obtaining the high-frequency and low-frequency data of the original mid-wave infrared image, and the high-frequency and low-frequency data of the original long-wave infrared image.

[0064] Next, for LatLRR, the L matrix needs to be trained on a long-wave and mid-wave dataset; based on this, in some implementations, the method further includes:

[0065] Based on the combination of the original mid-wave infrared image and the original long-wave infrared image, the L matrix is ​​obtained through training using the LatLRR expressed by the following formula:

[0066]

[0067] Wherein, the balance coefficient λ>0; ||·|| * The kernel norm is the sum of the singular values ​​of the matrix; ||·||1 is the l1 norm; X represents the combination of the original mid-wave infrared image and the original long-wave infrared image; Z represents the low-order coefficients; L represents the projection matrix, also known as the saliency coefficients; and E represents the sparse noise matrix.

[0068] It should be noted that after obtaining the high-frequency and low-frequency data, as well as the trained L matrix, the image reconstruction process can begin. To ensure the clarity of weak targets in the fused image and prevent the amplification of background noise, a salient component fusion strategy is employed. First, the image detail information is obtained by fusing the matrix L with the high-frequency results (salient components) from different bands. Based on this, image features are obtained through grayscale contrast modulation using the image detail information and the salient components.

[0069] Based on the above, in some implementations, fusing the salient part with the L matrix obtained through training the latent low-rank representation LatLRR to obtain detailed information includes:

[0070] According to the significant part D F And the detailed information D is obtained from the L matrix trained by LatLRR according to the following formula. E :

[0071] D E =L×D F .

[0072] Based on the above implementation, in some examples, obtaining salient features based on grayscale contrast modulation of the salient portion and the detailed information includes:

[0073] The significant part D F (i,j), perform two-dimensional filtering according to the following formula to obtain the filtered image D. F1 (i,j):

[0074] D F1 (i,j)=filter2(w,D F (i,j))

[0075] Here, filter2() represents a two-dimensional filtering operation; based on constraints such as symmetry, normalization type, and the contribution of odd and even terms, the template selected for the filtering window function is...

[0076] Based on the grayscale contrast formula shown below, through the significant portion D... F (i,j) and the filtered image D F1 Process (i,j) to obtain the contrast C(i,j):

[0077]

[0078] Based on the contrast C(i,j), the gray levels are discretized according to the following formula to obtain the new gray values ​​I(x,y):

[0079]

[0080] Among them, the maximum gray value of the image Minimum value

[0081] Based on the new grayscale value I(x,y) and the detailed information D E Perform convolution to obtain a new grayscale image I1(x,y);

[0082] The gradient operator is used to discretize I1(x,y) into grayscale values, and its grayscale values ​​are enhanced by the contrast method shown in the following formula to obtain a fused image I2(x,y) based on grayscale contrast that represents significant features:

[0083]

[0084] Where, I1(i,j) min I1(i,j) represents the minimum value of the new gray level. max This indicates the maximum value of the new grayscale level.

[0085] Based on the above example, fusing the salient features with the contour portion to obtain a reconstructed image includes:

[0086] Based on the salient feature I2(x,y) and the contour portion D L(x,y) The reconstructed image F(x,y) is obtained by fusing the data according to the following formula:

[0087] F(x,y)=I2(x,y)+D L(x,y) .

[0088] Based on the detailed explanations and examples provided above, the image fusion process can be described as follows: Figure 2 As shown, firstly, the LatLRR algorithm is used to train the L matrix on a long-wave and mid-wave infrared dataset to extract salient features. Next, the original long-wave and mid-wave infrared images are decomposed into high-frequency and low-frequency data using DWT. Then, the low- and high-frequency components of the long-wave infrared image are fused using maximum absolute value fusion to obtain the salient components; the low- and high-frequency components of the mid-wave infrared image are fused using a weighted average to obtain the contour components. Based on this, the trained matrix L and the salient components are fused using contrast modulation to obtain salient features. Finally, the reconstructed image is obtained by fusing the contour components and salient features.

[0089] After completing the image fusion stage, the next step is to detect small targets based on the reconstructed image. This invention proposes a U-Transformer model based on an encoder-decoder, taking into account the characteristics of small targets in infrared images, such as small scale, few features, and high noise. The reconstructed image extracts features and suppresses noise through the encoder, while simultaneously encoding the image to a lower dimension to ensure processing speed. Next, the image is encoded with relative positions and fed into the Transformer part for feature extraction. Compared to CNN models, the Transformer part has global modeling capabilities, thus effectively capturing the features of small targets. Finally, a high-quality feature map is obtained through the decoder, and the location information of the small target is predicted based on this map.

[0090] Based on this, in some possible implementations, the U-Transformer model includes a cascaded encoder part, a Transformer part, and a decoder part; wherein,

[0091] The encoder section includes multiple cascaded encoder modules for encoding the relative positions of the reconstructed image to preserve spatial information and obtain feature maps that the Transformer section can process.

[0092] The Transformer part is used to extract features of weak targets from the feature map;

[0093] The decoder section includes multiple cascaded decoders for generating feature maps based on the features of weak targets extracted by the Transformer section.

[0094] It should be noted that using an encoder to reduce the dimensionality of the reconstructed image, compared to directly introducing a Transformer network, reduces the computational load and effectively improves the real-time performance of the algorithm.

[0095] In some examples of the above implementation, the Transformer component includes multiple alternately cascaded W-TF and SW-TF modules, with a TF Down module positioned in the middle of the cascaded W-TF and SW-TF modules.

[0096] Based on the above implementation methods and examples, Figure 3Taking the U-Transformer model as an example, this model consists of two encoders, a Transformer (TF Module), and five decoders cascaded in sequence. Data transmission from the cascaded encoders to the TF Module can be downsampled as shown by the solid arrows; data transmission from the TF Module to the last decoder in the cascade can be upsampled as shown by the dashed arrows, for example, through interpolation.

[0097] For each encoder, such as Figure 3 As shown on the right, both can be composed of two layers of Conv convolution, Batch Normalization (BN), and ReLU activation function. These two encoders effectively suppress noise in the reconstructed image, highlight weak target features, and reduce the computational cost of subsequent operations by encoding into smaller feature maps. Furthermore, they can encode the relative position of the reconstructed image to preserve its spatial information, and can also encode it into a data format that the Transformer part can process before feeding it into the Transformer. Specifically, for the encoder's processing, the input shape of the reconstructed image is set to C. in_channel The ×H×W feature map is first resized to a suitable dimension using a convolution operation, with the encoding dimension C set. enbed_dim For example, with kernel size and stride both being P, non-overlapping convolutions are used to transform the above feature maps into... Then, the dimensions of the feature map are readjusted to transform it into a form that the Transformer part can process, i.e. Next, the transformed feature map is fed into the Transformer part.

[0098] For the Transformer part, multiple alternating cascaded W-TF modules, SW-TF modules, and TF Down modules, such as Figure 4 In the example shown, there are 5 W-TF modules and 5 SW-TF modules, with a TF Down module between the third SW-TF module and the fourth W-TF module; for each W-TF module or SW-TF module, combined with Figure 5 Taking the components shown as an example, the calculation process includes:

[0099] The transformed feature map described above Divided into A set of non-overlapping elements of size M×M×C enbed_dim The vector X, The only difference between the SW-TF and W-TF modules is the window partitioning method during MSA computation, specifically the local window feature X. First, it passes through a layer normalization layer; then, it calculates the multi-head self-attention (MSA) value within the window; finally, it passes through a... Figure 6 The MLP multilayer perceptron structure and formula shown The layer normalization process shown is illustrated.

[0100] The above calculation process can be expressed by the following formula:

[0101] In the MSA of the above formula, the matrices Q, K, and V corresponding to the query, key, and value in a specific window are respectively represented by Q = XP. Q K = XP K V = XP V Perform the calculation; where P Q ,P K ,P V These represent the shared matrices in different windows. Therefore, the attention value for each window can be obtained using the self-attention mechanism calculation formula shown below:

[0102]

[0103] Where B is a learnable relative position code.

[0104] In the decoder section, conventional interpolation methods can be used to achieve upsampling amplification. Compared to inverse convolution, interpolation can reduce the model size. Similarly, for each decoder... Figure 3 As shown on the right, they can all be composed of two layers of dilated convolution DConv, BN batch normalization, and ReLU activation function.

[0105] Based on the U-Transformer model described above and in the examples, in some examples, the step of performing target recognition on the reconstructed image based on the U-Transformer model to obtain the location information of weak targets includes:

[0106] After generating the feature map corresponding to the reconstructed image based on the U-Transformer model, the heat map of the target, the center offset of the target, and the scale information of the target are obtained through three parallel detections.

[0107] Based on the coordinates of the target with the highest confidence in the heat map, the position offset and width and height information corresponding to the weak target are extracted from the center offset and scale information of the target, and finally the position information of the weak target is obtained.

[0108] For the above example, it should be noted that the specific implementation process of generating the feature map corresponding to the reconstructed image based on the U-Transformer model corresponds to the aforementioned description of the U-Transformer model, and will not be repeated here. After obtaining the feature map, it can be fed into three different detection heads to obtain the target's heatmap, target center offset, and target scale information, respectively. Based on the coordinate point with the highest confidence in the target in the heatmap, the position offset and width / height information of the target in the corresponding detection head are extracted, ultimately forming the complete position information of the target.

[0109] Regarding the aforementioned technical solutions, their implementation methods, and examples, Figure 7 The example shown is for... Figure 7 The infrared long-wave image shown on the left and the infrared mid-wave image shown in the middle, after being processed according to the technical solution proposed in the embodiments of the present invention, yield the following detection results: Figure 7 The image on the right shows the final detection results of the small target from the original infrared image.

[0110] Based on the same inventive concept as the aforementioned technical solution, see [link to inventive concept]. Figure 8 This illustration shows a weak target detection device 80 based on infrared mid- and long-wave fusion according to an embodiment of the present invention. The device 80 includes: a wavelet transform decomposition part 801, a first fusion part 802, a second fusion part 803, a third fusion part 804, a modulation part 805, a fourth fusion part 806, and a recognition part 807; wherein,

[0111] The wavelet transform decomposition part 801 is configured to perform discrete wavelet transform decomposition on the original mid- and long-wave infrared image to obtain high-frequency data and low-frequency data respectively.

[0112] The first fusion portion 802 is configured to perform maximum absolute value fusion on the high-frequency data to obtain a significant portion;

[0113] The second fusion portion 803 is configured to perform weighted average fusion of the low-frequency data to obtain the contour portion;

[0114] The third fusion part 804 is configured to fuse the salient part with the L matrix obtained by training through the latent low-rank representation LatLRR to obtain detailed information;

[0115] The modulation section 805 is configured to obtain salient features by modulating based on grayscale contrast according to the salient portion and the detailed information;

[0116] The fourth fusion portion 806 is configured to fuse the salient features with the contour portion to obtain a reconstructed image;

[0117] The recognition section 807 is configured to perform target recognition on the reconstructed image based on a U-Transformer model that combines a U-shaped neural network and a Transformer algorithm, thereby obtaining the location information of weak targets.

[0118] For the above solutions, see some examples. Figure 9 The device 80 further includes a training section 808, configured to obtain an L matrix based on a combination of the original mid-wave infrared image and the original long-wave infrared image, using the LatLRR formula represented by the following equation:

[0119]

[0120] Wherein, the balance coefficient λ>0; ||·|| * The kernel norm is the sum of the singular values ​​of the matrix; ||·||1 is the l1 norm; X represents the combination of the original mid-wave infrared image and the original long-wave infrared image; Z represents the low-order coefficients; L represents the projection matrix, also known as the saliency coefficients; and E represents the sparse noise matrix.

[0121] In some examples of the above scheme, the third fusion portion 804 is configured as follows:

[0122] According to the significant part D F And the detailed information D is obtained from the L matrix trained by LatLRR according to the following formula. E :

[0123] D E =L×D F .

[0124] In some examples of the above scheme, the modulation section 805 is configured as follows:

[0125] The significant part D F (i,j), perform two-dimensional filtering according to the following formula to obtain the filtered image D. F1 (i,j):

[0126] D F1 (i,j)=filter2(w,D F (i,j))

[0127] Here, filter2() represents a two-dimensional filtering operation. Indicates the template used by the filtering window function;

[0128] Based on the grayscale contrast formula shown below, through the significant portion D... F (i,j) and the filtered image DF1 Process (i,j) to obtain the contrast C(i,j):

[0129]

[0130] Based on the contrast C(i,j), the gray levels are discretized according to the following formula to obtain the new gray values ​​I(x,y):

[0131]

[0132] Among them, the maximum gray value of the image Minimum value

[0133] Based on the new grayscale value I(x,y) and the detailed information D E Perform convolution to obtain a new grayscale image I1(x,y);

[0134] The gradient operator is used to discretize I1(x,y) into grayscale values, and its grayscale values ​​are enhanced by the contrast method shown in the following formula to obtain a fused image I2(x,y) based on grayscale contrast that represents significant features:

[0135]

[0136] Where, I1(i,j) min I1(i,j) represents the minimum value of the new gray level. max This indicates the maximum value of the new grayscale level.

[0137] In some examples of the above scheme, the fourth fusion portion 806 is configured as follows:

[0138] Based on the salient feature I2(x,y) and the contour portion D L(x,y) The reconstructed image F(x,y) is obtained by fusing the data according to the following formula:

[0139] F(x,y)=I2(x,y)+D L(x,y) .

[0140] In some examples of the above scheme, the U-Transformer model includes a cascaded encoder, a Transformer, and a decoder; wherein,

[0141] The encoder section includes multiple cascaded encoder modules for encoding the relative positions of the reconstructed image to preserve spatial information and obtain feature maps that the Transformer section can process.

[0142] The Transformer part is used to extract features of weak targets from the feature map;

[0143] The decoder section includes multiple cascaded decoders for generating feature maps based on the features of weak targets extracted by the Transformer section.

[0144] In some examples of the above scheme, the Transformer part includes multiple alternately cascaded W-TF modules and SW-TF modules, and a TF Down module is set in the middle of the cascaded W-TF modules and SW-TF modules.

[0145] In some examples of the above scheme, the identification part 807 is configured as follows:

[0146] After generating the feature map corresponding to the reconstructed image based on the U-Transformer model, the heat map of the target, the center offset of the target, and the scale information of the target are obtained through three parallel detections.

[0147] Based on the coordinates of the target with the highest confidence in the heat map, the position offset and width and height information corresponding to the weak target are extracted from the center offset and scale information of the target, and finally the position information of the weak target is obtained.

[0148] Understandably, in this embodiment, "part" can be a part of a circuit, a part of a processor, a part of a program or software, etc., or it can be a unit, a module, or a non-modular one.

[0149] Furthermore, in this embodiment, the components can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0150] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Therefore, this embodiment provides a computer storage medium storing a weak target detection program based on infrared mid- and long-wave fusion. When the weak target detection program based on infrared mid- and long-wave fusion is executed by at least one processor, it implements the steps of the weak target detection method based on infrared mid- and long-wave fusion described in the above technical solution.

[0152] Based on the aforementioned weak target detection device 80 based on infrared mid- and long-wave fusion and computer storage medium, see [link to relevant documentation]. Figure 10 This illustration shows the specific hardware structure of a computing device 100 capable of implementing the aforementioned weak target detection device 80 based on infrared mid-to-long-wave fusion, according to an embodiment of the present invention. The computing device 100 can be a wireless device, mobile or cellular phone (including so-called smartphones), personal digital assistant (PDA), video game console (including video display, mobile video game device, mobile video conferencing unit), laptop computer, desktop computer, set-top box, tablet computing device, e-book reader, fixed or mobile media player, etc. The computing device 100 includes: a communication interface 1001, a memory 1002, and a processor 1003; the various components are coupled together through a bus system 1004. It is understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 10 The general designated all buses as Bus System 1004.

[0153] The communication interface 1001 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0154] The memory 1002 is used to store computer programs that can run on the processor 1003;

[0155] The processor 1003 is used to execute the steps of the weak target detection method based on infrared mid- and long-wave fusion in the above technical solution when running the computer program.

[0156] It is understood that the memory 1002 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1002 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] The processor 1003 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1003 or by instructions in software form. The processor 1003 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1002. Processor 1003 reads the information in memory 1002 and completes the steps of the above method in conjunction with its hardware.

[0158] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0159] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0160] It is understood that the exemplary technical solutions of the infrared mid- and long-wave fusion-based weak target detection device 80 and computing device 100 described above belong to the same concept as the aforementioned weak target detection method based on infrared mid- and long-wave fusion. Therefore, all details not described in detail above regarding the technical solutions of the infrared mid- and long-wave fusion-based weak target detection device 80 and computing device 100 can be found in the description of the aforementioned weak target detection method based on infrared mid- and long-wave fusion. This embodiment of the invention will not elaborate further on these details.

[0161] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A small and weak target detection method based on infrared medium and long wave fusion, characterized in that, The method comprises: Discrete wavelet transform is performed on the original medium-long wave infrared image to obtain high-frequency data and low-frequency data; The high-frequency data are fused by maximum absolute value to obtain a salient part; The low-frequency data are fused by weighted average to obtain a contour part; The salient part is fused with an L matrix obtained by LatLRR training to obtain detail information; The salient part and the detail information are modulated based on gray contrast to obtain a salient feature; The salient feature is fused with the contour part to obtain a reconstructed image; An U-Transformer model combining a U-shaped neural network and a Transformer algorithm is used to perform target recognition on the reconstructed image to obtain position information of a small weak target; The U-Transformer model comprises a cascaded encoder part, a Transformer part and a decoder part; wherein The encoder part comprises a plurality of encoder modules connected in cascade, and is configured to perform relative position coding on the reconstructed image to retain spatial information and obtain a feature map that can be processed by the Transformer part; The Transformer part comprises a plurality of W-TF modules and SW-TF modules connected in alternation, and a TF Down module is arranged in the middle of the cascaded W-TF modules and SW-TF modules, and is configured to extract features of a small weak target from the feature map; The decoder part comprises a plurality of decoders connected in cascade, and is configured to generate a feature map according to the features of the small weak target extracted by the Transformer part.

2. The method of claim 1, wherein, The method further comprises: An L matrix is obtained by LatLRR training based on combination of an original medium wave infrared image and an original long wave infrared image according to the following formula: wherein the balance coefficient ; denotes the nuclear norm, which is the sum of the singular values of a matrix; is the norm; X denotes the combination of the inputted original mid-wave infrared image and the original long-wave infrared image; Z denotes the low-order coefficient; L denotes the projection matrix, which is called a significant coefficient; E denotes a sparse noise matrix.

3. The method of claim 1, wherein, The fusion of the salient part and the L matrix obtained by LatLRR training to obtain detail information comprises: According to the salient part and the LatLRR trained L matrix, the detail information is obtained according to the following formula : 。 4. The method of claim 3, wherein, The modulation of the salient part and the detail information based on gray contrast to obtain a salient feature comprises: said salient portion , a filtered image is obtained by two-dimensional filtering according to the following formula : wherein, represents a two-dimensional filtering operation, represents a template for selection of a filter window function; Based on a gray scale contrast formula shown in the following equation, the salient portion and the filtered image are processed to obtain a contrast : ； Based on contrast The gray level is discretized according to the following formula to obtain a new gray value : wherein the maximum value of the image gray scale the minimum value ; based on the new gray value with the detail information convolution is performed to obtain a new gray scale image ; using a gradient operator The gray scale is discretized, and the gray scale value is enhanced by a contrast method shown in the following equation to obtain a fusion image based on the contrast of the gray scale representing the salient features : wherein represents a new gray scale minimum value, represents a new gray scale maximum value.

5. The method of claim 4, wherein, The fusion of the salient feature and the contour part to obtain a reconstructed image comprises: According to the salient features and the contour portion , fusing according to the following formula, the reconstructed image is obtained : 。 6. The method of claim 1, wherein, The target recognition of the reconstructed image by the U-Transformer model combining a U-shaped neural network and a Transformer algorithm to obtain position information of a small weak target comprises: After a feature map corresponding to the reconstructed image is generated by the U-Transformer model, a heat map of a target, a center offset of the target and scale information of the target are obtained by three parallel detections; According to a coordinate point with the highest target confidence in the heat map, position offset and width-height information corresponding to the small weak target are extracted from the center offset of the target and the scale information of the target, and finally the position information of the small weak target is obtained.

7. A small dim target detection device based on infrared medium-long wave fusion, characterized in that, The device comprises a wavelet transform decomposition part, a first fusion part, a second fusion part, a third fusion part, a modulation part, a fourth fusion part and an identification part; wherein The wavelet transform decomposition part is configured to decompose the original mid-long wave infrared image by discrete wavelet transform to obtain high-frequency data and low-frequency data respectively; The first fusion part is configured to perform maximum absolute value fusion on the high-frequency data to obtain a salient part; The second fusion part is configured to perform weighted average fusion on the low-frequency data to obtain a contour part; The third fusion part is configured to fuse the salient part with an L matrix obtained by LatLRR training to obtain detailed information; The modulation part is configured to modulate the salient part and the detailed information based on gray contrast to obtain a salient feature; The fourth fusion part is configured to fuse the salient feature and the contour part to obtain a reconstructed image; The recognition part is configured to perform target recognition on the reconstructed image based on a U-Transformer model combined by a U-shaped neural network and a Transformer algorithm to obtain position information of a dim small target; The recognition part is further configured to the U-Transformer model including a cascaded encoder part, a Transformer part and a decoder part; wherein, The encoder part includes a plurality of encoder modules cascaded, which are used to encode the relative position of the reconstructed image to retain spatial information and obtain a feature map that can be processed by the Transformer part; The Transformer part includes a plurality of W-TF modules and SW-TF modules alternately cascaded, and a TF Down module is arranged in the middle of the cascaded W-TF modules and SW-TF modules, which is used to extract features of the dim small target from the feature map; The decoder part includes a plurality of decoders cascaded, which are used to generate a feature map according to the features of the dim small target extracted by the Transformer part.

8. A computer storage medium, characterized in that The computer storage medium stores a dim small target detection program based on infrared mid-long wave fusion, and the dim small target detection program based on infrared mid-long wave fusion is executed by at least one processor to realize the steps of the dim small target detection method based on infrared mid-long wave fusion in any one of claims 1 to 6.

Images