Infrared image super-resolution method, device and medium based on state space model

Abstract

The application discloses an infrared image super-resolution method based on a state space model, equipment and a medium. The method comprises the following steps: inputting a low-resolution infrared image into a shallow convolution layer, outputting initial image features through feature extraction, outputting fusion features fused with long-range dependence relationships through key information fusion, capturing thermal diffusion granularity features through multi-scale convolution, modeling through a state space equation, outputting state space features, performing feature enhancement, outputting enhanced state space features, aggregating long-range context information, outputting enhanced global features, performing local detail refinement, performing up-sampling through a sub-pixel convolution layer, and outputting a reconstructed high-resolution infrared image. While keeping low computational complexity, the method realizes high-fidelity infrared image super-resolution reconstruction and is suitable for application scenarios with high real-time requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, in particular to an infrared image super-resolution method based on a state space model, a device and a medium. BACKGROUND

[0002] Infrared image super-resolution technology is a research direction in the field of computer vision, and has high value in application scenarios such as military reconnaissance, medical diagnosis, and security monitoring. The core goal of this technology is to improve the quality of low-resolution infrared images and achieve high-precision reconstruction, including enhancing image details and suppressing noise.

[0003] In traditional schemes, the mainstream infrared image super-resolution method is mainly based on Convolutional Neural Network (CNN) and Vision Transformer (ViT) architecture.

[0004] However, in traditional schemes, the CNN-based method is limited by its local receptive field characteristics, making it difficult to effectively model the long-range spatiotemporal continuity caused by thermal diffusion effects in infrared images, which often leads to problems such as edge blurring and texture distortion in the reconstructed image. On the other hand, although the ViT-based method can capture global context information, its reliance on the self-attention mechanism has a computational complexity of O (N2), which increases memory consumption in high-resolution image processing scenarios, making it difficult to meet real-time processing application requirements.

[0005] At the same time, due to the significant thermal diffusion effect of infrared images, which makes the thermal signal exhibit continuity characteristics in the spatial dimension, traditional schemes generally fail to model this key characteristic, resulting in artifacts and loss of details in their reconstruction results. SUMMARY

[0006] To solve the above problems, the present application proposes an infrared image super-resolution method based on a state space model, comprising:

[0007] inputting a low-resolution infrared image into a shallow convolutional layer to output initial image features through feature extraction;

[0008] inputting the initial image features into a spatial attention module to output fusion features that fuse long-range dependencies through key information fusion;

[0009] inputting the fusion features into a gated state space module to capture thermal diffusion granularity features through multi-scale convolution and model them through a state space equation to output state space features;

[0010] ​inputting the state space feature into a nonlinear gate unit for feature enhancement, and outputting the state space feature after feature enhancement;

[0011] inputting the state space feature after feature enhancement as input, performing multi-level efficient state space group iteration processing, aggregating long-range context information, and outputting enhanced global feature;

[0012] inputting the global feature into a residual local feature block for local detail refinement, and performing up-sampling through a sub-pixel convolution layer, and outputting a reconstructed high-resolution infrared image.

[0013] In one example, the initial image feature is input into a spatial attention module, query vectors, key vectors and value vectors are generated in parallel through depth separable convolution, and cosine similarity between each pixel point corresponding to the initial image feature is calculated;

[0014] According to the cosine similarity, the key information corresponding to the highest key region is selected to obtain an attention weight matrix;

[0015] The attention weight matrix and the value vector corresponding to the initial image feature are input, and a fusion feature fused with long-range dependency is output through weighted aggregation.

[0016] In one example, the fusion feature is input into a gated state space module, and the fusion feature is subjected to convolution operation through convolution operation of multiple scales to obtain corresponding intermediate features;

[0017] The intermediate features corresponding to multiple scales are spliced to obtain multi-scale fusion features as thermal diffusion granularity features.

[0018] In one example, according to the size of the thermal diffusion granularity feature, the thermal diffusion granularity feature is reshaped into a feature sequence;

[0019] The state space model is obtained by modeling through a state space equation;

[0020] The feature sequence is input into the state space model, each element in the feature sequence is processed to obtain an output sequence, and the output sequence is reshaped to obtain a state space feature.

[0021] In one example, the state space feature is input into a nonlinear gate unit, and the state space feature is divided into a first feature branch and a second feature branch through channel segmentation;

[0022] For the first feature branch, a channel weight is learned through layer normalization, and an activation is performed through a Gaussian error linear unit to obtain a channel-adaptive weight vector;

[0023] The second feature branch and the channel adaptive weight vector are inputted, and the channel adaptive weight fusion is performed through element-by-element multiplication, and a state space feature after feature enhancement is outputted.

[0024] In one example, the state space feature after feature enhancement is inputted, and a data processing procedure corresponding to the spatial attention module, the gated state space module, and the nonlinear gating unit is sequentially performed through each level of the multi-level efficient state space group.

[0025] The output of the previous level of the efficient state space group is inputted into the next level of the efficient state space group, and an enhanced global feature is outputted through the last level of the efficient state space group.

[0026] In one example, the enhanced global feature is inputted into a residual local feature block, and high-frequency detail refinement and local information compensation are performed through multi-layer local convolution operation and nonlinear activation function, and a local feature after detail enhancement is obtained.

[0027] The local feature after detail enhancement is inputted, 4 times upsampling resolution reconstruction is performed through a sub-pixel convolution layer, and a reconstructed high-resolution infrared image is outputted.

[0028] In one example, through joint optimization training, a real high-resolution infrared image is taken as a supervision target, and model parameter training optimization is performed on at least part of the modules in the shallow convolution layer, the spatial attention module, the gated state space module, the nonlinear gating unit, the multi-level efficient state space group, and the sub-pixel convolution layer through an L1 loss function.

[0029] On the other hand, the present application also proposes an infrared image super-resolution device based on a state space model, comprising:

[0030] at least one processor; and

[0031] a memory in communication connection with the at least one processor; wherein

[0032] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the infrared image super-resolution method based on the state space model as described in any of the above examples.

[0033] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, and the computer executable instructions are set as the infrared image super-resolution method based on the state space model as described in any of the above examples.

[0034] The infrared image super-resolution method based on the state space model proposed in the present application can bring the following beneficial effects:

[0035] By introducing the spatial attention mechanism and the gated state space modeling, the long-range spatio-temporal dependence relationship caused by heat diffusion in the infrared image is effectively captured, and the edge blur and texture distortion problems caused by the limitation of the local receptive field of the CNN method are overcome.

[0036] The linear computational complexity advantage of the state space model is utilized to significantly reduce the memory consumption and computational overhead, solving the bottleneck of the real-time application of the ViT method in high-resolution image processing.

[0037] The heat diffusion granularity features are modeled by the multi-scale convolution and the state space equation, and the nonlinear gating enhancement and local detail refinement are combined, which significantly improves the detail reconstruction capability and reduces the artifacts and detail loss.

[0038] While maintaining low computational complexity, high-fidelity infrared image super-resolution reconstruction is achieved, which is suitable for application scenarios with high real-time requirements. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 The flowchart of the infrared image super-resolution method based on the state space model in the embodiments of the present application is shown;

[0041] Figure 2 The schematic diagram of the infrared image super-resolution device based on the state space model in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0043] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0044] As shown in Figure 1 The infrared image super-resolution method based on the state space model provided in the embodiments of the present application includes:

[0045] S101: input the low-resolution infrared image into a shallow convolutional layer, and output initial image features through feature extraction.

[0046] The low-resolution infrared image refers to an infrared image with a resolution lower than a preset degree, which is lower than the resolution of the high-resolution infrared image mentioned below.

[0047] Through shallow feature extraction, initial convolution operation is performed on the input low-resolution infrared image to extract initial image features, which can also be referred to as basic feature maps, which can be shown as Formula One:

[0048] Formula One;

[0049] wherein, is the low-resolution infrared image, is a convolution operation, and the size of the corresponding convolution kernel is 3x3, is the initial image feature.

[0050] S102: input the initial image feature into a spatial attention module, and output a fusion feature fused with long-range dependency through key information fusion.

[0051] Specifically, the initial image feature is input into the spatial attention module, and query vectors, key vectors and value vectors are generated in parallel through depth separable convolution, and the cosine similarity between each pixel point corresponding to the initial image feature is calculated. The spatial attention module can also be referred to as a spatial Top-k attention module, and dynamic attention fusion is performed through the spatial Top-k attention module.

[0052] According to the cosine similarity, the key information corresponding to the highest key region is selected to obtain an attention weight matrix. The first k key regions (key hot signal regions) are dynamically selected as the corresponding key information to reduce redundant calculation, and a corresponding feature map is output through weighted aggregation, which is shown as Formula Two and Formula Three:

[0053] Formula Two;

[0054] wherein, M is an attention weight matrix, is a depth separable convolution operation, and the size of the corresponding convolution kernel is 3x3, is the initial image feature, T is a matrix transposition operation, is the key vector.

[0055] Formula Three;

[0056] wherein, is a sparse attention weight matrix, an element in the i-th row and the j-th column of the attention weight matrix after sparsification, an element in the i-th row and the j-th column of the attention weight matrix, sort all elements in the j-th column of the attention weight matrix M, and select a set of the top k values with the largest values.

[0057] The attention weight matrix and the value vector corresponding to the initial image feature are input, and a fusion feature that integrates long-range dependencies is output through weighted aggregation. Through the dynamics of the spatial attention module, the key information interaction can be focused on, and the calculation efficiency can be improved. This is shown in Equation Four:

[0058] Equation Four;

[0059] wherein, is a fusion feature, is a normalization function, and each column of the attention weight matrix after sparsification is normalized so that the sum of the weights of all non-zero elements in each column is 1; is a convolution operation, and the size of the corresponding convolution kernel is 1×1. The initial image feature is convolved to generate a value vector.

[0060] S103: input the fusion feature into a gating state space module, capture thermal diffusion granularity features through multi-scale convolution, and model through a state space equation to output a state space feature.

[0061] Specifically, when capturing thermal diffusion granularity features, the fusion feature is input into the gating state space module, and the fusion feature is convolved through convolution operations of multiple scales to obtain corresponding intermediate features. The intermediate features corresponding to the multiple scales are spliced to obtain a multi-scale fusion feature as the thermal diffusion granularity feature, which can be shown in Equation Five:

[0062] Equation Five;

[0063] wherein, is a fusion feature, , , all are convolution operations, and the sizes of the corresponding convolution kernels are 1×1, 3×3, and 5×5; is a splicing operation, is a multi-scale fusion feature.

[0064] When modeling, the thermal diffusion granularity feature is reshaped into a feature sequence according to the size of the thermal diffusion granularity feature; Reshape the feature sequence with the size of [H, W, C] (height, width, channel) to a feature sequence with the length of L = H * W, which flattens the spatial dimension and gets a sequence with the length of L = H * W, and the feature dimension of each element in the sequence is C.

[0065] Modeling by state space equation to get a state space model, which can be shown in Equation Six and Equation Seven:

[0066] Equation Six

[0067] Equation Seven

[0068] where t is a discrete time step, which corresponds to the sequence position of the flattened image feature map in image processing, is the input at time t, which corresponds to the multi-scale fusion feature is the feature vector at sequence position t is the hidden state of the system at time t, which contains the memory of all historical information from the beginning of the sequence to time t is the hidden state of the system at time t+1 is the output at time t, which corresponds to the state space feature; A, B, C, D are state transition matrix, input matrix, output matrix, and feedforward matrix, respectively, which are all learnable parameter matrices.

[0069] Input the feature sequence into the state space model to process each element in the feature sequence to get an output sequence, and reshape the output sequence to get the state space feature.

[0070] After all elements in the sequence are processed, an output sequence with the same length L is obtained. The output sequence is reshaped back to the original spatial size [H, W, C'] to get the state space feature.

[0071] S104: Input the state space feature into the nonlinear gating unit for feature enhancement, and output the state space feature after feature enhancement.

[0072] Specifically, the state space feature is input into the nonlinear gating unit, and the state space feature is divided into a first feature branch and a second feature branch by channel segmentation, which can be shown in Equation Eight:

[0073] Equation Eight

[0074] wherein , are the first feature branch and the second feature branch, respectively, and the two branches have the same spatial size (height, width), but the number of channels is half of the state space feature y(t). for the channel segmentation operation, for the state space feature.

[0075] For the first feature branch, learn the channel weight through layer normalization and activate through Gaussian error linear unit to obtain the channel adaptive weight vector. Take the second feature branch and the channel adaptive weight vector as input, and perform channel adaptive weighted fusion through element-wise multiplication to output the enhanced state space feature. Element-wise multiplication is used to realize channel adaptive weighted fusion, highlight important channel features, suppress noise, further enhance the model's ability to represent infrared thermal dynamic characteristics, and enhance thermal dynamic adaptability. It can be shown as formula nine:

[0076] Formula nine;

[0077] wherein, is the enhanced state space feature, is a layer normalization operation, which normalizes each feature vector in the first feature branch to have a mean of 0 and a variance of 1; is a Gaussian error linear unit activation function, which introduces a smooth nonlinear transformation to the normalized feature; is an element-wise multiplication, which means multiplying the elements at corresponding positions of two matrices or tensors of the same dimension; Overall, it constitutes an adaptive gating signal. This signal contains context information from branch after nonlinear transformation.

[0078] S105: Take the enhanced state space feature as input, process it through multi-level efficient state space groups, aggregate long-range context information, and output enhanced global features.

[0079] Specifically, take the enhanced state space feature as input, and pass it through each level of the multi-level efficient state space group. The spatial attention module, the gated state space module, and the nonlinear gating unit perform data processing in sequence. The output of the previous level of the efficient state space group is taken as the input of the next level of the efficient state space group. The last level of the efficient state space group outputs the enhanced global features.

[0080] Further, taking a four-level efficient state space group as an example, it can be shown as formula ten:

[0081] Formula ten;

[0082] wherein, is the enhanced global feature, The enhanced state space feature is input into a residual local feature block for local detail refinement, and is up-sampled by a sub-pixel convolution layer to output a reconstructed high-resolution infrared image. The four-dimensional efficient state space group is repeatedly executed in each level of the efficient state space group, and the data processing processes of the space attention module, the gate state space module and the nonlinear gate unit are repeatedly executed. Through the inter-level jump connection or the feature fusion mechanism, the iterative aggregation and deepening of long-range context information are realized. The lower level captures the basic dependence, and the higher level integrates more abstract and more profound context information, and gradually improves the understanding of the complex heat diffusion mode.

[0083] S106: inputting the global feature into a residual local feature block for local detail refinement, and up-sampling by a sub-pixel convolution layer to output a reconstructed high-resolution infrared image.

[0084] Specifically, the enhanced global feature is input into a residual local feature block, and high-frequency detail refinement and local information (including image edges, textures and high-frequency details) compensation are performed through multi-layer local convolution operation and nonlinear activation function to obtain detail-enhanced local features. The detail-enhanced local features are input into a sub-pixel convolution layer for 4 times up-sampling resolution reconstruction, and a reconstructed high-resolution infrared image with enhanced details and artifact suppression is output.

[0085] The high-frequency detail refinement and local information compensation can be as shown in formula eleven:

[0086] Formula eleven;

[0087] Wherein, The detail-enhanced local feature is input into a residual local feature block, is a convolution operation, and the size of the corresponding convolution kernel is 3x3, is a rectified linear unit activation function, is an enhanced global feature.

[0088] The up-sampling resolution reconstruction can be as shown in formula twelve:

[0089] Formula twelve;

[0090] Wherein, is a high-resolution infrared image; is a pixel reorganization operation, which is realized by convolution + periodic screening to realize magnification. Assuming that the target is 4 times up-sampling, first expand the channel number to times through convolution, and then rearrange the feature map according to a specific rule, so that the spatial size (H, W) is expanded by 4 times, and the channel number is reduced to 1 / 16 of the original; is a convolution operation, and the size of the corresponding convolution kernel is 3x3.

[0091] By introducing a spatial attention mechanism and a gating state space model, the long-range spatio-temporal dependence relationship caused by heat diffusion in infrared images is effectively captured, overcoming the edge blurring and texture distortion problems caused by the limitation of the local receptive field of the CNN method.

[0092] By taking advantage of the linear computational complexity of the state space model, the memory consumption and computational overhead are significantly reduced, solving the bottleneck of real-time application of the ViT method in high-resolution image processing.

[0093] By modeling the heat diffusion granularity features through multi-scale convolution and state space equations, and combining nonlinear gating enhancement and local detail refinement, the detail reconstruction capability is significantly improved, and the artifacts and detail loss are reduced.

[0094] While maintaining low computational complexity, high-fidelity infrared image super-resolution reconstruction is achieved, which is suitable for application scenarios with high real-time requirements.

[0095] In one embodiment, during model training, through joint optimization training, the real high-resolution infrared image is taken as the supervision target, and the model parameter training optimization of at least part of the modules in the shallow convolution layer, the spatial attention module, the gating state space module, the nonlinear gating unit, the multi-level efficient state space group, and the sub-pixel convolution layer is performed through the L1 loss function. The loss function can be as shown in Equation Thirteen:

[0096] Equation Thirteen

[0097] wherein, is the reconstructed high-resolution infrared image, is the real high-resolution infrared image, is the L1 norm.

[0098] As Figure 2 shown, the embodiments of the present application also propose an infrared image super-resolution device based on a state space model, comprising:

[0099] at least one processor; and

[0100] a memory in communication connection with the at least one processor; wherein

[0101] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the infrared image super-resolution method based on the state space model as described in any of the above embodiments.

[0102] The embodiment of the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to implement the infrared image super-resolution method based on a state space model.

[0103] The embodiments in the present application are described in a progressive manner, and the same or similar parts of the embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, the device and medium embodiments are described simply because they are basically similar to the method embodiments, and the related parts can be referred to the part of the method embodiments.

[0104] The device and medium provided by the embodiments of the present application are one-to-one corresponding to the method, and therefore, the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0105] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0106] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0107] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0108] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0109] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0110] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, which can provide temporary storage for information. The memory can also include non-volatile memory, such as read-only memory (ROM) and / or flash memory, which can provide longer-term storage for information. The memory is an example of computer-readable media.

[0111] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology for storing information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carriers.

[0112] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0113] ​​The above merely provides an example of the present application, but is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. An infrared image super-resolution method based on a state space model, characterized in that, The method comprises the steps of: inputting a low-resolution infrared image into a shallow convolutional layer to output initial image features through feature extraction; inputting the initial image features into a spatial attention module to output fused features fused with long-range dependency relationships through key information fusion; specifically, inputting the initial image features into the spatial attention module, generating query vectors, key vectors and value vectors in parallel through depth separable convolution, and calculating the cosine similarity between each pixel point corresponding to the initial image features; selecting key information corresponding to the highest number of key regions according to the cosine similarity to obtain an attention weight matrix; taking the attention weight matrix and the value vector corresponding to the initial image features as inputs, and outputting fused features fused with long-range dependency relationships through weighted aggregation; inputting the fused features into a gated state space module to capture thermal diffusion granularity features through multi-scale convolution and modeling through a state space equation to output state space features; specifically, inputting the fused features into the gated state space module, and performing convolution operations on the fused features through convolution operations of multiple scales to obtain corresponding intermediate features; splicing the intermediate features corresponding to multiple scales to obtain multi-scale fused features as thermal diffusion granularity features; and reshaping the thermal diffusion granularity features into a feature sequence according to the size of the thermal diffusion granularity features; modeling through a state space equation to obtain a state space model; inputting the feature sequence into the state space model to process each element in the feature sequence to obtain an output sequence, and reshaping the output sequence to obtain state space features; inputting the state space features into a nonlinear gating unit for feature enhancement to output state space features after feature enhancement; inputting the state space features after feature enhancement as input, iteratively processing through multiple efficient state space groups to aggregate long-range context information, and outputting enhanced global features. inputting the state space features into a nonlinear gating unit for feature enhancement to output state space features after feature enhancement, specifically comprising:

2. The method of claim 1, wherein, inputting the state space features into a nonlinear gating unit to divide the state space features into a first feature branch and a second feature branch through channel segmentation; for the first feature branch, learning channel weights through layer normalization and activating through a Gaussian error linear unit to obtain a channel-adaptive weight vector; inputting the second feature branch and the channel-adaptive weight vector as input, performing channel-adaptive weighted fusion through element-wise multiplication to output state space features after feature enhancement. inputting the state space features after feature enhancement as input, iteratively processing through multiple efficient state space groups to aggregate long-range context information, and outputting enhanced global features, specifically comprising:

3. The method of claim 2, wherein, ​ The enhanced state space features are input into each high-efficiency state space group in the multi-level high-efficiency state space group, and the data processing process of the space attention module, the gated state space module, and the nonlinear gating unit is sequentially performed. The output of the previous high-efficiency state space group is input into the next high-efficiency state space group, and the enhanced global features are output by the last high-efficiency state space group.

4. The method of claim 1, wherein, The global features are input into a residual local feature block for local detail refinement, and are upsampled by a sub-pixel convolution layer to output a reconstructed high-resolution infrared image, including: The enhanced global features are input into a residual local feature block, and are subjected to high-frequency detail refinement and local information compensation by multi-layer local convolution operation and nonlinear activation function to obtain detail-enhanced local features. The detail-enhanced local features are input, and a 4-fold upsampled resolution reconstruction is performed by a sub-pixel convolution layer to output a reconstructed high-resolution infrared image.

5. The method of claim 1, wherein, The method further includes: Through joint optimization training, the real high-resolution infrared image is taken as a supervision target, and at least part of the modules in the shallow convolution layer, the space attention module, the gated state space module, the nonlinear gating unit, the multi-level high-efficiency state space group, and the sub-pixel convolution layer are subjected to model parameter training optimization by an L1 loss function.

6. An infrared image super-resolution device based on a state space model, characterized by, It includes: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the infrared image super-resolution method based on the state space model according to any one of claims 1-5.

7. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer to perform: The computer executable instructions are set as the infrared image super-resolution method based on the state space model according to any one of claims 1-5.

Images