A method and system for super-resolution reconstruction of infrared images based on visible light structure guidance

Abstract

A method and system for infrared image super-resolution reconstruction based on visible light structure guidance are disclosed. The infrared image super-resolution reconstruction method includes: acquiring a low-resolution infrared image and its corresponding high-resolution visible light image through preprocessing to construct a dataset; constructing an infrared image super-resolution reconstruction network, including a main feature reconstruction branch consisting of a shallow feature extraction module, a deep feature extraction module, and a final reconstruction module connected in series, and a structure prior guidance branch connected in parallel with the main feature reconstruction branch, consisting of a cross-modal structure calibration module; iteratively training the infrared image super-resolution reconstruction network using the dataset; deploying the trained infrared image super-resolution reconstruction network to a device and performing infrared image reconstruction to obtain the reconstruction result. This invention effectively utilizes visible light structure information and achieves accurate reconstruction of infrared image details through multi-scale structure prior guidance, thereby improving the quality of infrared image super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for super-resolution reconstruction of infrared images based on visible light structure guidance. Background Technology

[0002] Infrared imaging relies on the temperature difference and emissivity between the target and the background for detection. It works well even at night and in extremely low visibility conditions, and has a wide range of applications in target detection, security monitoring, and autonomous driving. However, due to limitations in hardware and cost, acquiring high-resolution infrared images is typically expensive and difficult. Existing low-resolution infrared images generally suffer from blurred details and poor texture information, which severely impacts the performance of subsequent downstream tasks. Therefore, developing efficient infrared image super-resolution reconstruction technology has significant research importance and practical application value.

[0003] In recent years, deep learning-based image super-resolution techniques have made significant progress, demonstrating powerful detail recovery capabilities in visible light image super-resolution reconstruction. However, when these methods are directly applied to infrared images, the reconstruction results are often unsatisfactory, and sometimes the model fails to function effectively. This is mainly because the imaging principle of infrared images results in characteristics such as poor texture, low contrast, and unclear structure, making it difficult to recover rich high-frequency details from their own information alone. To overcome the limitations of insufficient information in infrared images, utilizing other modal images, especially high-resolution visible light images, as guides to provide high-frequency texture and structural information for infrared super-resolution has become a very promising research direction in infrared image super-resolution, significantly improving the reconstruction results in terms of sharpness, edge sharpness, and detail integrity.

[0004] However, existing infrared image super-resolution methods based on visible light priors still suffer from the following problems: First, the fusion strategies of existing methods are simple, with most only directly fusing visible light features in the early or late stages using simple feature fusion methods, failing to effectively mine visible light structural information; second, the prior guidance mechanisms of existing methods lack specificity, making it difficult to achieve accurate selection and effective injection of guiding features, resulting in artifacts in the reconstructed image. Therefore, how to effectively integrate visible light structural information and achieve accurate injection of guiding features is currently a key direction in visible light-guided infrared image super-resolution reconstruction research. Summary of the Invention

[0005] This invention provides a method and system for super-resolution reconstruction of infrared images based on visible light structure guidance, in order to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention provides a method for super-resolution reconstruction of infrared images based on visible light structure guidance, comprising the following steps: S1. Preprocess the original infrared and visible light images to obtain low-resolution infrared images and corresponding high-resolution visible light images, and construct a dataset. S2. Construct an infrared image super-resolution reconstruction network based on visible light structure guidance, including a feature reconstruction main branch consisting of a shallow feature extraction module, a deep feature extraction module, and a final reconstruction module connected in series, and a structure prior guidance branch consisting of a cross-modal structure calibration module connected in parallel with the feature reconstruction main branch. S3. Use the dataset to iteratively train the infrared image super-resolution reconstruction network to obtain the trained infrared image super-resolution reconstruction network. S4. Deploy the trained infrared image super-resolution reconstruction network to the device and perform infrared image reconstruction to obtain the reconstruction results.

[0007] Furthermore, the shallow feature extraction module includes a 3×3 convolution, a ReLU activation function, and a 3×3 convolution connected in series. The cross-modal structure calibration module includes a parallel feature extraction branch and a structure calibration block connected in series; wherein the parallel feature extraction branch includes two parallel feature extraction sub-branches, each of which includes a Sobel gradient operator, a 3×3 convolution, and a feature enhancement block connected in series. The input and output of the feature enhancement block are connected via residuals. The feature enhancement block consists of a series of 3×3 convolutions, LeakyReLU activation functions, and 3×3 convolutions. The structure calibration block includes sequentially connected feature splicing, 1×1 convolution, two cascaded deep residual attention, 3×3 convolution, channel attention, and 3×3 convolution, with a residual connection introduced from the 3×3 convolution of one of the feature extraction sub-branches after the channel attention.

[0008] Furthermore, the deep feature extraction module adopts a U-Net structure, including a three-level encoder, a bottleneck layer, and a three-level decoder. In the encoding stage, the input features are sequentially extracted layer by layer by the three-level encoder, and the receptive field is expanded between each level of the encoder through downsampling operations. The bottleneck layer is located between the three-level encoder and the three-level decoder and is used to aggregate information from the deep features extracted by the last level encoder. In the decoding stage, the features processed by the bottleneck layer are sequentially reconstructed layer by layer by the three-level decoder, and the spatial resolution of the features is restored level by level by upsampling operations between each level of the decoder. The output features of each level encoder are fused with the input features of the corresponding level decoder through skip connections to compensate for the loss of spatial detail information caused by downsampling.

[0009] Furthermore, each level of the encoder, decoder, and bottleneck layer is composed of a structure-guided feature extraction module. The structure-guided feature extraction module contains m cascaded structure prior guidance modules and 3×3 convolutions. The input of the first structure prior guidance module and the output of the 3×3 convolution in the structure-guided feature extraction module are connected by a residual. The structure prior guidance module includes a first-layer normalization, a self-attention block, a second-layer normalization, a structure-guided cross-attention block, a third-layer normalization, and a multi-scale depth convolution feedforward block, which are connected in series. The inputs of the first, second, and third-layer normalization are respectively residually connected to the outputs of the self-attention block, the structure-guided cross-attention block, and the multi-scale depth convolution feedforward block.

[0010] Furthermore, step S3 specifically includes the following steps: S31. First, perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. Then, stitch the upsampled low-resolution infrared images with the corresponding high-resolution visible light images and use the result as input to the shallow feature extraction module to obtain the shallow feature map F. S32. Input the upsampled low-resolution infrared image and the high-resolution visible light image into the cross-modal structure calibration module, and use the visible light structure to perform adaptive calibration enhancement on the infrared structure to obtain the high-frequency structure calibration features of the infrared image. S33. In the deep feature extraction module, high-frequency structure calibration features are injected as multi-scale structure priors to mine multi-scale deep features and obtain the final feature map. S34. Input the final feature map into the final reconstruction module, and obtain the high-resolution reconstruction result of the final infrared image through 3×3 convolution. ; S35. Based on the high-resolution reconstruction results of the final infrared image Construct a loss function and adjust the infrared image super-resolution reconstruction network using the loss function; S36. Determine whether the set iteration stopping condition has been met. If yes, output the trained infrared image super-resolution reconstruction network; otherwise, return to S31.

[0011] Furthermore, step S31 specifically includes the following steps: S311. Perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. ; S312, Transfer the corresponding high-resolution visible light image Compared with upsampled low-resolution infrared images After concatenation, the data is input into the shallow feature extraction module. In the shallow feature extraction module, the data is processed by a series of 3×3 convolutions, ReLU activation functions, and 3×3 convolutions to output a shallow feature map F.

[0012] Furthermore, step S32 specifically includes the following steps: S321, First, upsample the low-resolution infrared image Corresponding high-resolution visible light image The input is fed into the cross-modal structure calibration module, where the dual inputs are mapped to the gradient domain using the Sobel gradient operator, and then the initial structural features are obtained through 3×3 convolution. , The expression is as follows: ; ; in, This represents the Sobel gradient operator. This represents a convolution with a kernel size of 3×3; S322, Initial structural features , The input is fed into the feature enhancement block to obtain the enhanced features; S323. The output enhanced features are concatenated, and channel fusion is achieved through 1×1 convolution. Then, adaptive structure calibration is performed sequentially through two cascaded levels of depth residual attention, 3×3 convolution, and channel attention. Finally, the output of the channel attention is combined with the initial structure features. Residual connections are used to obtain high-frequency structural calibration features of infrared images through 3×3 convolution. .

[0013] Furthermore, the structure-guided feature extraction module uses high-frequency structural calibration features injected into the infrared image. To achieve multi-scale deep feature mining, the operation process of the structure-guided feature extraction module is expressed as follows: ; in, and These represent the input and output of the structure-guided feature extraction module, respectively. This represents a cascaded structure prior guiding module.

[0014] Furthermore, the loss function in S35 is specifically as follows: ; Where N represents the number of training samples; This represents the i-th reconstructed high-resolution infrared image; This represents the corresponding true high-resolution infrared image of the i-th image; Indicates The pixel-level loss is calculated using the paradigm.

[0015] In another aspect, the present invention provides an infrared image super-resolution reconstruction system based on visible light structure guidance, configured to perform an infrared image super-resolution reconstruction method based on visible light structure guidance, comprising: The building unit is configured to build an infrared image super-resolution reconstruction network based on visible light structure guidance. The infrared image super-resolution reconstruction network includes a shallow feature extraction module, a cross-modal structure calibration module, a deep feature extraction module, and a final reconstruction module. The shallow feature extraction unit is configured to obtain a shallow feature map F using the shallow feature extraction module; The cross-modal structure calibration unit is configured to use the cross-modal structure calibration module to adaptively calibrate and enhance the infrared structure using the visible light structure in the gradient domain, thereby obtaining the high-frequency structure calibration features of the infrared image. The deep feature extraction unit is configured to use the deep feature extraction module to inject high-frequency structure calibration features as multi-scale structure priors, mine multi-scale deep features, and obtain the final feature map. The final reconstruction unit is configured to use 3×3 convolution to obtain a high-resolution reconstruction result of the final infrared image. .

[0016] The beneficial effects of this invention are: 1. The cross-modal structure calibration module in this invention can use high-resolution visible light images to calibrate the structural features of infrared images, obtain reliable high-frequency structural calibration features of infrared images, and use them as structural priors to help low-resolution infrared images recover clearer edge structures and details.

[0017] 2. The structure prior guidance module in this invention can achieve accurate injection of structure priors and make full use of high-frequency structure calibration features to highlight and restore the edge structure and details of infrared images. Attached Figure Description

[0018] Figure 1 This is a flowchart of the infrared image super-resolution reconstruction method in this invention; Figure 2 This is a block diagram of the infrared image super-resolution reconstruction network in this invention; Figure 3 This is a block diagram of the cross-modal structure calibration module; Figure 4 The structural block diagram of the structural prior guidance module; Figure 5 This is a schematic diagram of the reconstruction result in an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0020] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] To address the aforementioned issues of blurred details in infrared images, the inability of existing visible light-guided fusion strategies to effectively extract visible light structural information, and the lack of targeted guidance mechanisms, a method and system for infrared image super-resolution reconstruction based on visible light structure guidance is proposed to facilitate the reconstruction of infrared image details. The specific steps of this method are as follows: Reference Figure 1 and Figure 2 This application provides a method for super-resolution reconstruction of infrared images based on visible light structure guidance, comprising the following steps: S1. Preprocess the original infrared image and visible light image to obtain a low-resolution infrared image and a corresponding high-resolution visible light image, and construct a dataset. The low resolution in the low-resolution infrared image refers to a resolution not higher than a first set threshold. In this embodiment, the first set threshold is set to 128*128 PPI (pixels per inch), but it can also be set to other values, such as 64*64 PPI. The high resolution in the high-resolution visible light image refers to a resolution not lower than a second set threshold. In this embodiment, the second set threshold is set to 256*256 PPI. The same provisions apply to low resolution and high resolution in other places. S2. Construct an infrared image super-resolution reconstruction network based on visible light information assistance, including a feature reconstruction main branch consisting of a shallow feature extraction module, a deep feature extraction module, and a final reconstruction module connected in series, and a structure prior guidance branch consisting of a cross-modal structure calibration module connected in parallel with the feature reconstruction main branch. S3. Use the dataset to iteratively train the infrared image super-resolution reconstruction network to obtain the trained infrared image super-resolution reconstruction network. S4. Deploy the trained infrared image super-resolution reconstruction network to the device and perform infrared image reconstruction to obtain the reconstruction results.

[0022] In some embodiments, refer to Figure 2 The shallow feature extraction module includes a series of 3×3 convolutions, a ReLU activation function, and another 3×3 convolution. Reference Figure 2 and Figure 3 The cross-modal structure calibration module includes a parallel feature extraction branch connected in series and a structure calibration block; wherein the parallel feature extraction branch includes two parallel feature extraction sub-branches, each of which includes a Sobel gradient operator, a 3×3 convolution, and a feature enhancement block connected in series. The input and output of the feature enhancement block are connected via residuals. The feature enhancement block consists of a series of 3×3 convolutions, LeakyReLU activation functions, and 3×3 convolutions. The structural calibration block includes sequentially connected feature splicing, 1×1 convolution, two cascaded deep residual attention, 3×3 convolution, channel attention, and 3×3 convolution, and introduces a residual connection from the 3×3 convolution of one of the feature extraction sub-branches after the channel attention. In some embodiments, refer to Figure 2 and Figure 4 The deep feature extraction module adopts a U-Net structure, including a three-level encoder, a bottleneck layer, and a three-level decoder. During the encoding stage, the input features are sequentially extracted layer by layer by the three-level encoder, with downsampling operations used between each level of the encoder to expand the receptive field. The bottleneck layer, located between the three-level encoder and the three-level decoder, is used to aggregate information from the deep features extracted by the last level encoder. During the decoding stage, the features processed by the bottleneck layer are sequentially reconstructed layer by layer by the three-level decoder, with upsampling operations used between each level of the decoder to restore the spatial resolution of the features. The output features of each level encoder are fused with the input features of the corresponding level decoder through skip connections to compensate for the loss of spatial detail information caused by downsampling.

[0023] In some embodiments, each encoder, decoder, and bottleneck layer is composed of a structure-guided feature extraction module. The structure-guided feature extraction module includes m cascaded structure prior guidance modules and 3×3 convolutions. The input of the first structure prior guidance module and the output of the 3×3 convolution in the structure-guided feature extraction module are connected by residuals. In this embodiment, the number of cascaded structure prior guidance modules is four.

[0024] In some embodiments, S3 specifically includes the following steps: S31. First, perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. Then, stitch the upsampled low-resolution infrared images with the corresponding high-resolution visible light images and use the result as input to the shallow feature extraction module to obtain the shallow feature map F. S32. Input the upsampled low-resolution infrared image and the high-resolution visible light image into the cross-modal structure calibration module, and use the visible light structure to adaptively calibrate and enhance the infrared structure in the gradient domain to obtain the high-frequency structure calibration features of the infrared image. S33. In the deep feature extraction module, high-frequency structure calibration features are injected as multi-scale structure priors to mine multi-scale deep features and obtain the final feature map. S34. Input the final feature map into the final reconstruction module, and obtain the high-resolution reconstruction result of the final infrared image through 3×3 convolution. ; S35. Based on the high-resolution reconstruction results of the final infrared image Construct a loss function and adjust the infrared image super-resolution reconstruction network using the loss function; S36. Determine whether the set iteration stopping condition has been met. If yes, output the trained infrared image super-resolution reconstruction network; otherwise, return to S31.

[0025] In some embodiments, S31 specifically includes the following steps: S311. Perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. ; S312, Transfer the corresponding high-resolution visible light image Compared with upsampled low-resolution infrared images After concatenation, the data is input into the shallow feature extraction module. In the shallow feature extraction module, the data undergoes a series of 3×3 convolutions, ReLU activation functions, and 3×3 convolutions to output a shallow feature map F.

[0026] In some embodiments, S32 specifically includes the following steps: S321, First, upsample the low-resolution infrared image Corresponding high-resolution visible light image The input is fed into the cross-modal structure calibration module, where the dual inputs are mapped to the gradient domain using the Sobel gradient operator to highlight high-frequency structural information. Then, the initial structural features are obtained through 3×3 convolution. , The expression is as follows: ; ; in, This represents the Sobel gradient operator. This represents a convolution with a kernel size of 3×3; S322. Then, feature enhancement is performed through feature enhancement blocks. Specifically, the initial structural features are first... , The input is fed into the feature enhancement block for feature enhancement. Feature mining is performed through the residual structure composed of two 3×3 convolutions and two LeakyReLU activation functions to obtain enhanced features. S323. Finally, structural calibration is performed using a structural calibration block. Specifically, the output enhanced features are first concatenated, and channel fusion is achieved through 1×1 convolution. Then, adaptive structural calibration is performed sequentially through two cascaded levels of depth residual attention, 3×3 convolution, and channel attention. Finally, the output of the channel attention is compared with the initial structural features. Residual connections are used to obtain high-frequency structural calibration features of infrared images through 3×3 convolution. .

[0027] In some embodiments, the structure-guided feature extraction module extracts high-frequency structural calibration features from an injected infrared image. To achieve multi-scale deep feature mining, the operation process of the structure-guided feature extraction module is expressed as follows: ; in, and This represents the input and output of the structure-guided feature extraction module; This represents a cascaded structure prior guiding module. This represents a convolution with a kernel size of 3×3.

[0028] In some embodiments, the structure prior guidance module includes a first-layer normalization, a self-attention block, a second-layer normalization, a structure-guided cross-attention block, a third-layer normalization, and a multi-scale deep convolutional feedforward block, connected in series. The inputs of the first, second, and third-layer normalization are residually connected to the outputs of the self-attention block, the structure-guided cross-attention block, and the multi-scale deep convolutional feedforward block, respectively. This invention achieves feature aggregation through self-attention, injects structure prior through structure-guided cross-attention, and further compensates for high-frequency information loss through the multi-scale deep convolutional feedforward block. The corresponding formulas are as follows: ; in, This represents the original input features, i.e., the input features of the structural prior guidance module. Indicates intermediate features, Indicates enhanced features, This represents the output characteristics of the structural prior guidance module. Representation layer normalization, Represents a self-attention block. This indicates a structure-guided cross-attention block. This represents a multi-scale depthwise convolution feedforward block.

[0029] In some embodiments, structure-guided cross-attention blocks use intermediate features. High-frequency structural calibration features of infrared images To achieve cross-attention, utilize intermediate features Extract query matrix Utilizing high-frequency structural calibration features from infrared images Extracting the key matrix Sum matrix The calculation formula is as follows: ; ; in, It is the softmax activation function. , , For learnable projection matrices, It is the feature dimension parameter. This represents the weighted features that guide cross-attention based on structure.

[0030] In some embodiments, multi-scale depthwise convolutional feedforward blocks can effectively compensate for the loss of high-frequency information caused by the attention mechanism, obtaining more details through multi-scale depthwise convolution. Specifically, feature enhancement. The system sequentially performs 1×1 convolution, channel segmentation, and parallel depthwise convolutions with kernel sizes of 1×1, 3×3, 5×5, and 7×7. Local features are mined through multi-scale receptive fields. The local feature channels are concatenated and connected to the output residual of the 1×1 convolution. Then, the first and second components are obtained through channel segmentation. The first component is enhanced with the GELU activation function to improve its nonlinear representation ability and is then multiplied element-wise with the second component. The output features are obtained through 1×1 convolution, which is the output feature of the structural prior guidance module.

[0031] In some embodiments, the loss function in S35 is specifically: ; Where N represents the number of training samples; This represents the i-th reconstructed high-resolution infrared image; This represents the corresponding true high-resolution infrared image of the i-th image; Indicates The pixel-level loss is calculated using the paradigm.

[0032] In another aspect, the present invention provides an infrared image super-resolution reconstruction system based on visible light structure guidance, configured to perform an infrared image super-resolution reconstruction method based on visible light structure guidance, comprising: The building unit is configured to build an infrared image super-resolution reconstruction network based on visible light structure guidance. The infrared image super-resolution reconstruction network includes a shallow feature extraction module, a cross-modal structure calibration module, a deep feature extraction module, and a final reconstruction module. The shallow feature extraction unit is configured to obtain a shallow feature map F using the shallow feature extraction module; The cross-modal structure calibration unit is configured to use the cross-modal structure calibration module to adaptively calibrate and enhance the infrared structure using the visible light structure in the gradient domain, thereby obtaining the high-frequency structure calibration features of the infrared image. The deep feature extraction unit is configured to use the deep feature extraction module to inject high-frequency structure calibration features as multi-scale structure priors, mine multi-scale deep features, and obtain the final feature map. The final reconstruction unit is configured to use 3×3 convolution to obtain a high-resolution reconstruction result of the final infrared image. .

[0033] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for super-resolution reconstruction of infrared images based on visible light structure guidance, characterized in that, Includes the following steps: S1. Preprocess the original infrared and visible light images to obtain low-resolution infrared images and corresponding high-resolution visible light images, and construct a dataset. S2. Construct an infrared image super-resolution reconstruction network based on visible light structure guidance, including a feature reconstruction main branch consisting of a shallow feature extraction module, a deep feature extraction module, and a final reconstruction module connected in series, and a structure prior guidance branch consisting of a cross-modal structure calibration module connected in parallel with the feature reconstruction main branch. S3. Use the dataset to iteratively train the infrared image super-resolution reconstruction network to obtain the trained infrared image super-resolution reconstruction network. S4. Deploy the trained infrared image super-resolution reconstruction network to the device and perform infrared image reconstruction to obtain the reconstruction results. The shallow feature extraction module includes a 3×3 convolution, a ReLU activation function, and a 3×3 convolution connected in series. The cross-modal structure calibration module includes a parallel feature extraction branch and a structure calibration block connected in series; wherein the parallel feature extraction branch includes two parallel feature extraction sub-branches, each of which includes a Sobel gradient operator, a 3×3 convolution, and a feature enhancement block connected in series. The input and output of the feature enhancement block are connected via residuals. The feature enhancement block consists of a series of 3×3 convolutions, LeakyReLU activation functions, and 3×3 convolutions. The structural calibration block includes sequentially connected feature splicing, 1×1 convolution, two cascaded deep residual attention, 3×3 convolution, channel attention, and 3×3 convolution, and introduces a residual connection from the 3×3 convolution of one of the feature extraction sub-branches after the channel attention. The deep feature extraction module adopts a U-Net structure, including a three-level encoder, a bottleneck layer, and a three-level decoder. During the encoding stage, the input features are sequentially extracted layer by layer by the three-level encoder. The receptive field is expanded between each level of the encoder through downsampling operations. The bottleneck layer is located between the three-level encoder and the three-level decoder and is used to aggregate information from the deep features extracted by the last level encoder. During the decoding stage, the features processed by the bottleneck layer are sequentially reconstructed layer by layer by the three-level decoder. The spatial resolution of the features is restored step by step between the levels of the decoder through upsampling operations. The output features of each level encoder are fused with the input features of the corresponding level decoder through skip connections to compensate for the loss of spatial detail information caused by downsampling. Each level of the encoder, decoder and bottleneck layer is composed of a structure-guided feature extraction module. The structure-guided feature extraction module contains m cascaded structure prior guidance modules and 3×3 convolutions. The input of the first structure prior guidance module and the output of the 3×3 convolution in the structure-guided feature extraction module are connected by residuals. The structure prior guidance module includes a first-layer normalization, a self-attention block, a second-layer normalization, a structure-guided cross-attention block, a third-layer normalization, and a multi-scale depth convolution feedforward block, which are connected in series. The inputs of the first, second, and third-layer normalization are respectively residually connected to the outputs of the self-attention block, the structure-guided cross-attention block, and the multi-scale depth convolution feedforward block.

2. The infrared image super-resolution reconstruction method based on visible light structure guidance according to claim 1, characterized in that, S3 specifically includes the following steps: S31. First, perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. Then, stitch the upsampled low-resolution infrared images with the corresponding high-resolution visible light images and use the result as input to the shallow feature extraction module to obtain the shallow feature map F. S32. Input the upsampled low-resolution infrared image and the high-resolution visible light image into the cross-modal structure calibration module, and use the visible light structure to perform adaptive calibration enhancement on the infrared structure to obtain the high-frequency structure calibration features of the infrared image. S33. In the deep feature extraction module, high-frequency structure calibration features are injected as multi-scale structure priors to mine multi-scale deep features and obtain the final feature map. S34. Input the final feature map into the final reconstruction module, and obtain the high-resolution reconstruction result of the final infrared image through 3×3 convolution. ; S35. Based on the high-resolution reconstruction results of the final infrared image Construct a loss function and adjust the infrared image super-resolution reconstruction network using the loss function; S36. Determine whether the set iteration stopping condition has been met. If yes, output the trained infrared image super-resolution reconstruction network; otherwise, return to S31.

3. The infrared image super-resolution reconstruction method based on visible light structure guidance according to claim 2, characterized in that, S31 specifically includes the following steps: S311. Perform bicubic interpolation upsampling on the low-resolution infrared images in the dataset to obtain upsampled low-resolution infrared images. ; S312, Transfer the corresponding high-resolution visible light image Compared with upsampled low-resolution infrared images After concatenation, the data is input into the shallow feature extraction module. In the shallow feature extraction module, the data is processed by a series of 3×3 convolutions, ReLU activation functions, and 3×3 convolutions to output a shallow feature map F.

4. The infrared image super-resolution reconstruction method based on visible light structure guidance according to claim 3, characterized in that, S32 specifically includes the following steps: S321, First, upsample the low-resolution infrared image Corresponding high-resolution visible light image The input is fed into the cross-modal structure calibration module, where the dual inputs are mapped to the gradient domain using the Sobel gradient operator, and then the initial structural features are obtained through 3×3 convolution. , The expression is as follows: ； ； in, This represents the Sobel gradient operator. This represents a convolution with a kernel size of 3×3; S322, Initial structural features , The input is fed into the feature enhancement block to obtain the enhanced features; S323. The output enhanced features are concatenated, and channel fusion is achieved through 1×1 convolution. Then, adaptive structure calibration is performed sequentially through two cascaded levels of depth residual attention, 3×3 convolution, and channel attention. Finally, the output of the channel attention is combined with the initial structure features. Residual connections are used to obtain high-frequency structural calibration features of infrared images through 3×3 convolution. .

5. The infrared image super-resolution reconstruction method based on visible light structure guidance according to claim 4, characterized in that, The structure-guided feature extraction module uses high-frequency structure calibration features injected into the infrared image. To achieve multi-scale deep feature mining, the operation process of the structure-guided feature extraction module is expressed as follows: ； in, and These represent the input and output of the structure-guided feature extraction module, respectively. This represents a cascaded structure prior guiding module.

6. The infrared image super-resolution reconstruction method based on visible light structure guidance according to claim 5, characterized in that, The loss function in S35 is specifically as follows: ； Where N represents the number of training samples; This represents the i-th reconstructed high-resolution infrared image; This represents the corresponding true high-resolution infrared image of the i-th image; Indicates The pixel-level loss is calculated using the paradigm.

7. A super-resolution reconstruction system for infrared images based on visible light structure guidance, characterized in that, Configured to perform the infrared image super-resolution reconstruction method based on visible light structure guidance as described in any one of claims 1 to 6, comprising: The building unit is configured to build an infrared image super-resolution reconstruction network based on visible light structure guidance. The infrared image super-resolution reconstruction network includes a shallow feature extraction module, a cross-modal structure calibration module, a deep feature extraction module, and a final reconstruction module. The shallow feature extraction unit is configured to obtain a shallow feature map F using the shallow feature extraction module; The cross-modal structure calibration unit is configured to use the cross-modal structure calibration module to adaptively calibrate and enhance the infrared structure using the visible light structure in the gradient domain, thereby obtaining the high-frequency structure calibration features of the infrared image. The deep feature extraction unit is configured to use the deep feature extraction module to inject high-frequency structure calibration features as multi-scale structure priors, mine multi-scale deep features, and obtain the final feature map. The final reconstruction unit is configured to utilize 3×3 convolution to obtain a high-resolution result of the final reconstructed infrared image. .

Images