Multi-band infrared image arbitrary multiplying power super-resolution method and system

Abstract

The invention discloses a multi-band infrared image arbitrary multiplying power super-resolution method and system, and aims to improve the reconstruction quality of an infrared image under different bands, complex degradation and multi-scale amplification conditions. The method comprises the following steps: constructing a degeneration and down-sampling image data set covering three wavebands of SWIR, MWIR and LWIR; designing an image and zoom factor combined prompt embedding mechanism for guiding the network to carry out adaptive affine regulation and control; constructing a joint reconstruction structure formed by connecting an image restoration sub-network and a super-division backbone network in series; and high-resolution reconstruction of an infrared image with any magnification in a reasoning stage is supported. The method and the system have the characteristics of high wave band adaptability, support of any magnification amplification, abundant degradation modeling, suitability for video stream reasoning deployment and the like, and are suitable for scenes of infrared imaging enhancement, video perception, unmanned platform visual processing and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a multi-band infrared image arbitrary-magnification super-resolution method and system. BACKGROUND

[0002] Infrared images have strong imaging robustness in complex lighting environments and are widely used in night vision monitoring, military reconnaissance, border warning, industrial detection and other scenes. According to the different wavelength ranges, infrared images are mainly divided into three bands: short-wave infrared (SWIR), medium-wave infrared (MWIR) and long-wave infrared (LWIR), each of which has different imaging mechanisms and image quality characteristics. Among them, the SWIR image is close to the visible light band, with clear texture but sensitive to low light environment; the MWIR image has good thermal contrast, suitable for detecting medium-distance heat sources; the LWIR image has strong penetration, suitable for long-distance imaging, but is easily affected by thermal convection, atmospheric absorption and sensor interference, and the image quality is severely degraded.

[0003] Due to physical limitations of sensors such as hardware resolution, pixel size, heat dissipation design, etc., actual infrared images generally have low resolution, thermal diffusion blur, Gaussian blur, sensor noise, and stripe interference, etc. These complex degradations not only reduce the visual quality of the image, but also bring great interference to subsequent recognition and detection tasks. In order to improve the image clarity and structural fidelity, image super-resolution reconstruction has become an important technical approach to improve the resolution of infrared images.

[0004] However, most of the current super-resolution methods are developed and trained on visible light image datasets, and these methods do not fully consider the significant differences in infrared image sensing mechanisms, blur patterns, texture expression methods, etc. Directly migrating visible light super-resolution models to the infrared image field often leads to problems such as unrealistic texture reconstruction and enhancement failure. Even if some methods are specifically designed for infrared images, they often only deal with single-band images, lack cross-band adaptability, and are difficult to cope with the complex degradation situation of blur, noise and stripe superposition in actual images.

[0005] In addition, in order to adapt to various display terminals and application requirements, in recent years, arbitrary-magnification super-resolution methods have appeared, which can realize arbitrary enlargement of images of any size. Most of these models are based on dynamic computation graph design and have flexibility, but in mobile terminal, embedded device and other scenarios, they are limited by the support ability of inference engine for dynamic graph structure, and are difficult to be directly deployed and used, which has obvious engineering gap.

[0006] In this context, prompt learning is introduced as a conditional regulation mechanism to dynamically guide the model to perform different task paths. Although this mechanism shows good flexibility in natural language processing and multi-modal tasks, current prompt forms mostly rely on natural language or manual label input, which is not suitable for real-time infrared application scenarios such as unmanned systems and video stream processing. Therefore, the prompt learning mechanism needs to be converted to an image feature driven adaptive prompt mechanism, so that the model can complete dynamic control according to the image itself and task requirements without relying on external language input.

[0007] Notably, the composite degradation in infrared images has the characteristics of uneven spatial distribution and variable type combination, and it is difficult to effectively model and restore various degradation contents by simply relying on a single super-resolution network. Therefore, an image restoration sub-network is introduced as a preprocessing path to specifically restore non-structural degradation such as blur, noise, and stripes, and is jointly trained with the super-resolution backbone network, which not only improves the input quality of image structural information, but also helps the model to learn robust adaptation to real complex degradation conditions.

[0008] In summary, there is currently a lack of a general infrared image super-resolution scheme that can adapt to multi-band infrared images, has composite degradation restoration capability, supports arbitrary magnification, and can be deployed in a static inference environment. Therefore, it is urgent to build an adaptive prompt mechanism based on image and magnification factor, and a unified method combining image restoration and super-resolution joint optimization strategy to comprehensively improve the super-resolution reconstruction quality and deployment feasibility of infrared images in real application scenarios. SUMMARY

[0009] The present application aims to overcome the problems of poor waveband adaptability, complex degradation not modeled, fixed magnification ratio, and lack of high-level guidance in existing infrared image super-resolution methods, and proposes a multi-band arbitrary magnification infrared image super-resolution method and system based on adaptive prompt learning and joint image restoration mechanism, which realizes adaptive perception of different infrared waveband images, image magnification of arbitrary scale, and collaborative recovery of complex degradation, thereby improving image quality and reconstruction accuracy.

[0010] To achieve the above purpose, the present application constructs an adaptive prompt learning mechanism based on image and magnification factor to obtain the implicit semantic information of image waveband characteristics and magnification requirements, and injects it into the backbone network as an affine regulation factor to realize task-driven super-resolution reconstruction. On the other hand, a pre-trained image deblurring network is introduced to be jointly trained with the super-resolution backbone network, which simultaneously improves the structural clarity and detail restoration ability through feature sharing and multi-task optimization.

[0011] The purpose of the present application is achieved by the following technical solutions:

[0012] In one aspect, the application discloses a multi-band infrared image arbitrary ratio super-resolution method, which comprises the following steps:

[0013] Step one, multi-band characteristic infrared image dataset construction, comprising the following sub-steps:

[0014] S1.1 Collect high-definition infrared images covering SWIR, MWIR and LWIR three types of bands;

[0015] S1.2 Generate a dataset for pre-training of the image restoration sub-network: based on different bands, design a degradation strategy, construct image samples simulating multiple degradation types, generate degraded image-high-definition image training pairs, and record the degradation type label;

[0016] S1.3 Generate a dataset for pre-training of the super-resolution main network: add light blur to the high-definition image, perform multi-ratio down-sampling and record the scaling factor, adjust the image size to a fixed size, and generate an interpolated low-resolution image-high-definition image training pair;

[0017] S1.4 Generate a dataset for joint training: combine the degradation and down-sampling operations to generate multi-ratio image samples with composite degradation, and record their band type, degradation type and scaling factor;

[0018] Step two, prompt embedding generation, comprising the following sub-steps:

[0019] S2.1 Input the degraded image into the image encoder to extract a global image feature vector containing band and degradation semantic information;

[0020] S2.2 Encode the continuous scaling factor value into a scaling factor vector of the same dimension for scale control through the factor encoder;

[0021] S2.3 Fuse the global image feature vector and the scaling factor vector through element-wise addition to generate a joint prompt embedding;

[0022] S2.4 In the training stage, a multi-label classification supervision branch of band and degradation type is added at the end of the image encoder;

[0023] Step three, joint training, comprising the following sub-steps:

[0024] S3.1 The image restoration sub-network adopts the Restormer structure, and an affine transformation structure controlled by the prompt is embedded after each Transformer module of the decoder;

[0025] S3.2 The super-resolution main network is based on SwinIR, introduces a compression attention and neighborhood attention gate fusion mechanism, and embeds an affine transformation structure controlled by the prompt after each Transformer module.

[0026] S3.3 Use the dataset constructed in step S1.2 to pre-train the image complex atom network, and use the dataset constructed in step S1.3 to pre-train the super-resolution backbone network.

[0027] S3.4 Connect the pre-trained image complex atom network and the super-resolution backbone network in a cascade structure to form a joint network, and train the joint network using the dataset constructed in step 1.4;

[0028] Step 4: High-definition reconstruction of multi-band infrared images at arbitrary magnification.

[0029] Furthermore, in step S1.2, the degradation strategy designed based on different bands is specifically as follows:

[0030] For SWIR degradation modeling, the original high-resolution image is first processed using a 3×3 Gaussian convolution kernel with a standard deviation of 0.6. Then, Gaussian noise with a mean of 0 and a standard deviation of 0.005, as well as speckle noise with a standard deviation of 0.1, are added to the processed image. Finally, contrast compression is performed using a gamma mapping with a gamma factor of 1.5.

[0031] For MWIR degradation modeling, a 3×3 mean convolution kernel is first used, Gaussian noise with a mean of 0 and a standard deviation of 0.01 is added, and dead points are randomly inserted into the image; then, the overall brightness of the processed image is moderately attenuated.

[0032] For LWIR degradation modeling, a 5×5 motion blur kernel is first used for processing, with random orientation and a length of 7 pixels; then periodic stripe noise interference is added to the processed image, and horizontal or vertical sine function perturbation terms are added.

[0033] For each degraded image, record the degradation type label it contains. The labels are stored in a multi-label format, and the image's band type label is also recorded.

[0034] Furthermore, in step S1.3, a downsampled image with multiple scaling factors is constructed from each high-definition image. Specifically, a slight blur is first added, then bicubic interpolation is used to achieve multi-fold downsampling, the downsampling scaling factor is recorded, and finally bicubic interpolation is used uniformly to adjust the image size to a fixed size.

[0035] Further, in step S1.3, each original high-resolution image is downsampled to generate 8 low-resolution image pairs with different magnifications, including fixed magnifications of ×2 and ×4, and the remaining 6 magnifications are randomly generated within the interval (2,4). Before all downsampling, a standard Gaussian kernel is used for slight blurring, and then bicubic interpolation is used to complete the reduction operation. The resulting low-resolution images are uniformly adjusted to the fixed input size required by the super-resolution backbone network through bicubic interpolation. The scaling factor is calculated before the second interpolation and saved as a prompt signal, while the band type label of the image is recorded.

[0036] Furthermore, in step two, the image encoder employs a shallow convolutional network, inputting the degraded image into the image encoder and outputting a global image feature vector; the factor encoder employs a multilayer perceptron, inputting continuous scaling factors as real scalars into the factor encoder, and performing nonlinear encoding through a multilayer perceptron containing two fully connected layers. Output a one-dimensional vector, expand this one-dimensional vector along the spatial dimension to match the dimension of the global image feature vector through a broadcast mechanism, and perform element-wise addition with the image feature vector to generate a joint cue embedding representation.

[0037] Furthermore, in step two, the multi-label classification supervision branch includes two parallel fully connected layers, which are used to predict the band type and degradation type of the image, respectively. During the training phase, the cross-entropy loss function and the multi-label binary cross-entropy loss function are used for supervision.

[0038] Further, in step three, the specific implementation process of the affine transformation structure controlled by the prompt is as follows: the joint prompt embedding representation is processed by a parameter generation module containing two parallel fully connected layers to generate the scale and shift required for the affine transformation, respectively.

[0039] The l-th layer in the Transformer module of the image complex atom network and super-resolution backbone network is represented as follows:

[0040]

[0041] in, For the first Intermediate features of the layer The intermediate features of layer l are updated after embedding the affine transformation structure as follows:

[0042]

[0043] in, This represents element-wise multiplication. and These represent the ratio and bias parameters of the l-th layer, respectively.

[0044] Furthermore, in step three, the super-resolution backbone network is constructed based on SwinIR, and the original attention module is replaced with a gated fusion structure of compressed attention and neighborhood attention;

[0045] The compressed attention mechanism is as follows: the key / value features are divided into blocks within a local window and compressed and mapped to obtain a low-dimensional compressed representation through depthwise separable convolution. The query features maintain full resolution and are subject to attention calculation. The neighborhood attention mechanism is as follows: each query feature is only subject to attention calculation with the key / value features in its k×k neighborhood.

[0046] The gating fusion mechanism is as follows: the query features are input into the multilayer perceptron to generate gating weights of dimension 2. After being normalized by Softmax, these weights are applied to the compressed attention and neighborhood attention outputs respectively, and then fused into the final attention response.

[0047] Furthermore, in step three, the image complex atom network and the super-resolution backbone network are pre-trained first, and then the network connections are jointly trained. Specifically, the image complex atom network and the super-resolution backbone network are connected in series to form a joint network. No additional processing is performed on the output of the image complex atom network, which is directly used as the input of the super-resolution backbone network. The image encoder part retains the parameters obtained from the training of the image complex atom network, and the factor encoder part retains the parameters of the super-resolution backbone network. In the joint training stage, the dataset constructed in step S1.4 is used, and L1 loss is used for supervised optimization. The optimizer and learning rate strategy are consistent with those in the pre-training stage of the super-resolution backbone network.

[0048] Another aspect of the present invention discloses a multi-band infrared image arbitrary magnification super-resolution system, the system comprising:

[0049] The multi-band infrared image data acquisition and construction module is used to acquire high-definition infrared images of different bands, construct training sets, validation sets and test sets containing different degradation types and low-resolution images at multiple magnifications, and record the band type label, degradation type label and scaling factor information of the corresponding images.

[0050] The embedded generation module includes an image encoder and a factor encoder, which are used to extract global semantic features and scaling factor features of the image, and output the scaling and bias values ​​through feature fusion and affine transformation structure to achieve cross-task adaptive control.

[0051] The super-resolution and restoration joint reconstruction network module consists of a joint network composed of an image complex atom network and a super-resolution backbone network. The image complex atom network is based on the Restormer structure and is responsible for handling multiple types of degradation. The super-resolution backbone network is based on the SwinIR structure and performs detail enhancement and resolution improvement. Both sub-networks embed affine transformation structures controlled by cueing, which are used to perform feature adaptive transformation according to image semantics and scaling requirements.

[0052] The inference and output module receives infrared images of arbitrary resolution and a given magnification factor, performs frame-by-frame or batch processing through a joint network, and outputs high-resolution images with clear texture and consistent magnification. It is suitable for real-time deployment scenarios of infrared video streams or unmanned systems.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. Achieving highly adaptable super-resolution reconstruction across multiple bands: This invention designs a differentiated degradation modeling and joint restoration mechanism for multiple infrared imaging bands such as SWIR, MWIR, and LWIR, and uses a cue branch for degradation perception, significantly improving the model's adaptability to image quality differences in different bands.

[0055] 2. Supports image super-resolution magnification at arbitrary magnification: By constructing a cue learning mechanism based on magnification factor, the network can adaptively magnify images according to any input magnification without the need for fixed-scale training or switching between multiple models, thus meeting the accuracy requirements of different tasks.

[0056] 3. Robust handling capability for complex degradation: The joint architecture of image complex atom network and super-resolution backbone network is introduced to effectively deal with common complex degradation conditions such as blurring, noise and stripes in infrared images, and improve the quality and stability of the final reconstructed image.

[0057] 4. Introducing an adaptive prompt learning mechanism that does not require manual annotation: This invention uses image features and magnification factors to jointly construct prompt embeddings, which do not rely on natural language or manual labels, support automatic reasoning in video streams or unmanned platform deployments, and have good real-time performance and versatility. Attached Figure Description

[0058] Figure 1 A flowchart of a method for super-resolution of multi-band infrared images at arbitrary magnification provided in an embodiment of the present invention;

[0059] Figure 2 A block diagram illustrating the construction and implementation of the multi-band feature infrared image dataset provided in this embodiment of the invention;

[0060] Figure 3 A block diagram illustrating the implementation of the prompt embedding generation and the affine transformation structure controlled by the prompt, provided in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram illustrating the implementation of image restoration and super-resolution according to an embodiment of the present invention;

[0062] Figure 5 This is a block diagram of a multi-band infrared image arbitrary magnification super-resolution system provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0064] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0065] like Figure 1 As shown in the figure, the multi-band infrared image arbitrary magnification super-resolution method provided in this embodiment has the following steps:

[0066] Step 1: Construct a multi-band characteristic infrared image dataset, such as... Figure 2 As shown, it includes the following sub-steps:

[0067] S1.1 uses a high-definition cooled infrared camera to acquire high-quality infrared images, covering three bands: SWIR, MWIR, and LWIR.

[0068] S1.2 For training the image complex atom network, a degradation strategy is designed based on different bands. Image samples simulating various degradation types (such as Gaussian blur, defocus blur, thermal diffusion, noise interference, etc.) are constructed to generate "degraded image - high-definition image" training pairs and record the degradation type labels.

[0069] S1.3 For training the super-resolution backbone network, a downsampled image with multiple scaling factors is constructed from each high-resolution image. First, a slight blur is added, and then bicubic interpolation is used to achieve multi-fold downsampling. The scaling factor of downsampling is recorded. Finally, bicubic interpolation is used to adjust the image size to a fixed size, and a training pair of "interpolated low-resolution image - high-resolution image" is constructed.

[0070] S1.4 For joint training, combine the above degradation and downsampling operations to generate composite degradation multi-magnification image samples, and record their band type, degradation type and scaling factor;

[0071] S1.5 uses non-HD infrared cameras of three bands to collect real scene images as the source of test input samples for the system inference phase.

[0072] The specific implementation details are as follows:

[0073] a) High-definition cooled infrared cameras specifically include the following three types of devices, covering the short-wave infrared (SWIR), mid-wave infrared (MWIR), and long-wave infrared (LWIR) bands respectively: SWIR image acquisition devices are indium gallium arsenide focal plane detectors with a spectral response band of 0.9μm~1.7μm and a camera resolution of 640×512; MWIR image acquisition devices are mercury cadmium telluride cooled infrared focal plane detectors with a spectral response band of 3.7μm~4.8μm and a camera resolution of 1280×1024; LWIR image acquisition devices are type II superlattice cooled infrared focal plane detectors with a spectral response band of 7.7~9.4μm and a camera resolution of 1280×1024.

[0074] b) For SWIR degradation modeling, the original high-resolution image I is first processed using a 3×3 Gaussian convolution kernel, with the standard deviation... Set to 0.6, the processed image is denoted as ;

[0075] Then add a mean of 0 and a standard deviation of 0. Gaussian noise and standard deviation The speckle noise is generated as follows:

[0076]

[0077] pass Contrast compression is performed using gamma mapping, in the following way:

[0078]

[0079] in, Gamma factor, This is the degraded image.

[0080] For MWIR degradation modeling, a 3×3 mean convolution kernel is used, Gaussian noise with a mean of 0 and a standard deviation of 0.01 is added, and dead pixels are randomly inserted into the image to obtain the final image. This means forcing pixel values ​​to 0 or the maximum grayscale value; for the image The overall brightness is moderately attenuated, specifically by using linear scaling to simulate thermal signal attenuation:

[0081]

[0082] in, This is the linear scaling factor.

[0083] For LWIR degradation modeling, a 5×5 motion blur kernel is used to obtain the image. The direction is random, and the length is 7 pixels; periodic stripe noise interference is added, and a horizontal or vertical sine function perturbation term is added. The formula for the vertical case is as follows:

[0084]

[0085] in, The noise amplitude, For noise period, This is a bias term.

[0086] For each degraded image, record the degradation type label (such as "Gaussian noise", "motion blur", "striped noise interference", etc.). The labels are stored in a multi-label format, and the band type label of the image is also recorded to support classification-supervised training in subsequent cue branches.

[0087] c) Each original high-resolution image I is downsampled to generate 8 pairs of low-resolution images with different magnifications, including fixed magnifications of ×2 and ×4, and the remaining 6 magnifications are randomly generated within the interval (2,4). All downsampling is performed using a standard Gaussian kernel (…). A slight blurring process is applied, followed by bicubic interpolation to reduce the image size. The resulting low-resolution image is then uniformly adjusted to the fixed input size required by the super-resolution backbone network using bicubic interpolation. The scaling factor r is calculated before the second interpolation and saved as a prompt signal, while the band type label of the image is recorded.

[0088] d) In step S1.4, based on the processing methods of steps S1.2 and S1.3, composite degradation is first constructed for high-definition images of different bands, followed by multi-rate downsampling. The degradation type is randomly selected from at least one type among Gaussian blur, motion blur, thermal diffusion blur, sensor noise, and stripe interference, with each degradation type being randomly selected. Simultaneously, the band type label, degradation type label (supporting multiple labels), and scaling factor value are recorded for each image sample.

[0089] e) The non-HD infrared cameras in the three bands are all infrared cameras with a resolution of no more than 640×512.

[0090] Step two, prompting for embedding generation, such as Figure 3 As shown in (a), it includes the following sub-steps:

[0091] S2.1 Input the degraded image into the image encoder to extract the global image feature vector containing band and degradation semantic information;

[0092] S2.2 The continuous scaling factor value S is encoded into a scaling factor vector of the same dimension for scale control by the factor encoder.

[0093] S2.3 The global image feature vector obtained in step S2.1 and the scaling factor vector obtained in step S2.2 are fused together by element-wise addition to generate a joint cue embedding;

[0094] S2.4 During the training phase, in order to improve semantic representation capabilities, a multi-label classification supervision branch for bands and degradation types is added to the end of the image encoder.

[0095] The specific implementation details are as follows:

[0096] a) The image encoder uses a shallow convolutional network, which refers to a network with three 3×3 convolutional layers. Let the input degraded image be... , Let the height, width, and channel dimensions of the image be respectively, then the image encoder... The calculation process is as follows:

[0097]

[0098] in,

[0099]

[0100] in, This represents a non-linear activation function, such as ReLU.

[0101] The image encoder outputs the feature vector as follows:

[0102]

[0103] b) The continuous scaling factor is input to the factor encoder in real scalar form, i.e. The factor encoder uses a multilayer perceptron (MLP). Nonlinear encoding is performed using a multilayer perceptron with a two-layer fully connected structure:

[0104]

[0105] Output a one-dimensional vector of length C, i.e. .

[0106] c) The image feature vector F output by the image encoder has dimensions H×W×C, and the continuous scaling factor, after being encoded by the multilayer perceptron, becomes a one-dimensional vector of length C. The vector is broadcast through a broadcast mechanism. Expanding along spatial dimensions:

[0107]

[0108] Then, element-wise addition is performed with the image feature vector to generate a joint cue embedding representation. .

[0109] d) The multi-label classification supervision branch consists of two parallel fully connected layers. The first branch is used for band type prediction (SWIR / MWIR / LWIR, single-label classification), and the output... The second branch is used for degradation type prediction (such as noise, fuzziness, multiple degradation forms, and multi-label classification), and the output is... The corresponding supervised loss function is:

[0110]

[0111] Among them, h Let cross-entropy be the loss function. This is the multi-label binary cross-entropy loss function. The final total loss is:

[0112]

[0113] in, This is the balance coefficient.

[0114] Step 3, joint training, includes the following sub-steps:

[0115] The S3.1 image complex atom network adopts a Restormer structure, embedding cue-controlled affine transformation structures after each Transformer module of the decoder. This structure is as follows: Figure 3 As shown in (b);

[0116] The S3.2 super-resolution backbone network is built upon an improved SwinIR architecture, introducing a fusion mechanism of compressed attention and neighborhood attention gating. Furthermore, a cue-controlled affine transformation structure is embedded after each Transformer module. This structure is as follows: Figure 3 As shown in (b);

[0117] S3.3 Using the data constructed in steps 1.2 and 1.3, pre-train the image complex atom network and the super-resolution backbone network respectively;

[0118] S3.4 The pre-trained image complex atom network and the super-resolution backbone network are connected in a cascade structure to form a joint network, such as... Figure 4 As shown, the joint network is trained using the data constructed in step 1.4.

[0119] The specific implementation details are as follows:

[0120] a) The image complex atom network adopts the Restormer structure, which belongs to the U-Net class architecture. It includes two parts: encoder and decoder. Only after each Transformer module of the decoder is an affine transformation structure controlled by cue embedded, which is used for feature adjustment in the decoding stage based on the semantics of image degradation type.

[0121] b) The super-resolution backbone network is built based on SwinIR, replacing the original attention module with a gated fusion structure of compressed attention and neighborhood attention, specifically:

[0122] Compressed attention mechanism: Let the input features be... First, the Query, Key, and Value are obtained through linear transformation or 1×1 convolution:

[0123]

[0124] in, Here, is a learnable parameter, and d is the feature dimension of the Query Key Value vector. Then... The local window is divided into blocks (window size M×M), and compression mapping is performed using depthwise separable convolution:

[0125]

[0126] in, This represents a convolution operation with stride *s*, used to reduce spatial resolution and achieve low-dimensional compressed representation. The formula for calculating the compressed attention output is:

[0127]

[0128] Neighborhood attention mechanism:

[0129] Unlike standard global attention, each query vector only performs attention calculations on its spatially adjacent k×k range of key / value features. The specific calculation formula is as follows:

[0130]

[0131] in, and This represents a subset of Key and Value features corresponding to the neighborhood of the Query's location. The Query, Key, and Value for neighborhood attention are shared with those for compressed attention.

[0132] Gating fusion mechanism:

[0133] To enable adaptive fusion between different attention branches, this invention introduces a gating mechanism. For each query feature q, two normalized gating weights are generated using a multilayer perceptron (MLP). and The contribution ratios of compressed attention and neighborhood attention are controlled separately:

[0134]

[0135] The final attention output is represented as:

[0136]

[0137] in, Softmax normalization is used to ensure a balanced weight distribution.

[0138] This gated fusion structure can dynamically adjust the contributions of compressed attention and neighborhood attention, achieving a balance between global low-rank representation and local detail modeling, thereby improving the quality of super-resolution reconstruction while ensuring computational efficiency.

[0139] c) The specific implementation process of the affine transformation structure controlled by prompts is as follows:

[0140] Union prompt embedding representation The parameter generation module (PGM), consisting of two parallel fully connected (FC) layers, generates the scale and shift values ​​required for the affine transformation.

[0141]

[0142]

[0143] in, This is the weight matrix of the fully connected layer. This is the corresponding bias term for the fully connected layer.

[0144] In the Transformer module of image complex atom networks and super-resolution backbone networks, the first... A layer can be represented as:

[0145]

[0146] in, For the first Intermediate features of the layer For the first The intermediate features of the layer. After embedding the affine transformation structure, the equation is updated to:

[0147]

[0148] in, This represents element-wise multiplication. and The first The proportions and bias parameters of the layers (each Transformer module corresponds to an independent parameter generation module PGM).

[0149] d) The image complex atom network is trained using the “degraded image – high-resolution image” training pair constructed in step S1.2 as training samples. L1 loss and AdamW optimizer are used. The training iterations are 300K times, the initial learning rate is 3e−4, and cosine annealing strategy is used for decay.

[0150] The super-resolution backbone network is trained using the “interpolated low-resolution image – high-resolution image” training pair constructed in step S1.3 as training samples. It adopts L1 loss and Adam optimizer, and trains for 500K iterations. The initial learning rate is 2e−4, which is halved in the [250K, 400K, 450K, 475K]th iteration.

[0151] e) The image complex atom network and the super-resolution backbone network are connected in series to form a joint network. No additional processing is performed on the output of the image complex atom network; it is directly used as the input to the super-resolution backbone network. In the cue embedding, the image encoder retains the parameters obtained from training the image complex atom network, and the factor encoder retains the parameters of the super-resolution backbone network. During the joint training phase, the composite degraded multi-fold image samples constructed in step S1.4 are used as training samples. L1 loss is used for supervised optimization, and the training iterations are performed for 500K times. The optimizer and learning rate strategy are consistent with those used in the pre-training phase of the super-resolution backbone network.

[0152] Step four: High-resolution reconstruction of multi-band infrared images at arbitrary magnification. Specific implementation details are as follows:

[0153] a) Acquire or read frame-by-frame images of infrared images or video streams to be processed, supporting raw image format input in three bands: SWIR, MWIR, and LWIR;

[0154] b) Perform resolution analysis on the input image. The target magnification factor (such as ×2.3, ×3.0, etc.) can be set by the user or automatically estimated according to the task requirements. The input image is then adjusted to a uniform network input resolution through bicubic interpolation to ensure consistency with the input size during the training phase.

[0155] c) Input the resized image into the joint network trained in step three to output a high-resolution image.

[0156] Another aspect of this invention discloses a multi-band infrared image arbitrary magnification super-resolution system, such as... Figure 4 As shown, the system includes:

[0157] The multi-band infrared image data acquisition and construction module is used to acquire high-definition infrared images of different bands, construct training sets, validation sets, and test sets containing different degradation types and low-resolution images at multiple magnifications, and record the band type label, degradation type label, and scaling factor information of the corresponding images; the implementation of this module can refer to step one above.

[0158] The embedded generation module includes an image encoder and a factor encoder, which are used to extract global semantic features and scaling factor features of the image. It outputs the scaling and bias values ​​through feature fusion and affine transformation structure to achieve cross-task adaptive control. The implementation of this module can refer to step two above.

[0159] The super-resolution and restoration joint reconstruction network module consists of a joint network composed of an image complex atom network and a super-resolution backbone network connected in series. The image complex atom network is based on the Restormer structure and is responsible for handling multiple types of degradation, while the super-resolution backbone network is based on the SwinIR structure and performs detail enhancement and resolution improvement. Both sub-networks embed affine transformation structures controlled by prompts, which are used to perform feature adaptive transformation according to image semantics and scaling requirements. The implementation of this module can refer to step three above.

[0160] The inference and output module is used to receive infrared images of arbitrary resolution and a given magnification factor, and to perform frame-by-frame or batch processing through the above-mentioned joint network to output high-resolution images with clear texture and consistent magnification. It is suitable for real-time deployment scenarios of infrared video streams or unmanned systems. The implementation of this module can refer to step four above.

[0161] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0162] This invention addresses the high-quality perception and enhancement tasks of multi-band infrared images. By constructing an adaptive cueing learning mechanism and an image restoration-super-resolution joint network, it solves the problems of poor band adaptability, lack of complex degradation modeling, and fixed magnification in traditional infrared image super-resolution methods. It achieves high-fidelity, degradation-robust, and detail-enhancing super-resolution reconstruction of input images at any magnification, thereby improving the visual quality and intelligent application capabilities of infrared images in complex environments.

[0163] The above description is merely a preferred embodiment of the present invention. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for super-resolution of multi-band infrared images at arbitrary magnification, characterized in that, The method includes: Step 1, constructing a multi-band characteristic infrared image dataset, includes the following sub-steps: S1.1 Acquire high-resolution infrared images covering three bands: SWIR, MWIR, and LWIR; S1.2 Generate a dataset for pre-training the image complex atom network: Design degradation strategies based on different bands, construct image samples simulating various degradation types, generate degraded image-high-resolution image training pairs, and record degradation type labels; S1.3 Generate a dataset for pre-training the super-resolution backbone network: add a slight blur to the high-resolution image, perform multi-fold downsampling and record the scaling factor, adjust the image size to a fixed size, and generate interpolated low-resolution image – high-resolution image training pairs. S1.4 Generate a dataset for joint training: Combine degradation and downsampling operations to generate composite degradation multi-magnification image samples and record their band type, degradation type and scaling factor; Step two, prompting for embedding generation, includes the following sub-steps: S2.1 Input the degraded image into the image encoder to extract the global image feature vector containing band and degradation semantic information; S2.2 The continuous scaling factor values ​​are encoded into a scaling factor vector of the same dimension for scale control by the factor encoder; S2.3 The global image feature vector and the scaling factor vector are fused through element-wise addition to generate a joint cue embedding; S2.4 During the training phase, a multi-label classification supervision branch for bands and degradation types is added to the end of the image encoder; Step 3, joint training, includes the following sub-steps: The S3.1 image complex atom network adopts the Restormer structure, embedding cue-controlled affine transformation structures after each Transformer module of the decoder; The S3.2 super-resolution backbone network is built on SwinIR, introduces a fusion mechanism of compressed attention and neighborhood attention gating, and embeds a cue-controlled affine transformation structure after each Transformer module; S3.3 Use the dataset constructed in step S1.2 to pre-train the image complex atom network, and use the dataset constructed in step S1.3 to pre-train the super-resolution backbone network. S3.4 Connect the pre-trained image complex atom network and the super-resolution backbone network in a cascade structure to form a joint network, and train the joint network using the dataset constructed in step 1.4; Step 4: High-definition reconstruction of multi-band infrared images at arbitrary magnification.

2. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step 1.2, the degradation strategy designed based on different bands is as follows: For SWIR degradation modeling, the original high-resolution image is first processed using a 3×3 Gaussian convolution kernel with a standard deviation of 0.

6. Then, Gaussian noise with a mean of 0 and a standard deviation of 0.005, as well as speckle noise with a standard deviation of 0.1, are added to the processed image. Finally, contrast compression is performed using a gamma mapping with a gamma factor of 1.

5. For MWIR degradation modeling, a 3×3 mean convolution kernel is first used, Gaussian noise with a mean of 0 and a standard deviation of 0.01 is added, and dead points are randomly inserted into the image; then, the overall brightness of the processed image is moderately attenuated. For LWIR degradation modeling, a 5×5 motion blur kernel is first used for processing, with random orientation and a length of 7 pixels; then periodic stripe noise interference is added to the processed image, and horizontal or vertical sine function perturbation terms are added. For each degraded image, record the degradation type label it contains. The labels are stored in a multi-label format, and the image's band type label is also recorded.

3. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step S1.3, a downsampled image with multiple scaling factors is constructed from each high-definition image. Specifically, a slight blur is first added, then bicubic interpolation is used to achieve multi-fold downsampling, the downsampling scaling factor is recorded, and finally bicubic interpolation is used to adjust the image size to a fixed size.

4. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 3, characterized in that, In step 1.3, each original high-resolution image is downsampled to generate 8 pairs of low-resolution images with different magnifications, including fixed magnifications of ×2 and ×4, and the remaining 6 magnifications are randomly generated within the interval (2,4). Before all downsampling, a standard Gaussian kernel is used for slight blurring, and then bicubic interpolation is used to complete the downsizing operation. The resulting low-resolution images are uniformly adjusted to the fixed input size required by the super-resolution backbone network through bicubic interpolation. The scaling factor is calculated before the second interpolation and saved as a prompt signal, and the band type label of the image is recorded at the same time.

5. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step two, the image encoder uses a shallow convolutional network, inputting the degraded image into the image encoder and outputting a global image feature vector; the factor encoder uses a multilayer perceptron, inputting continuous scaling factors as real scalars into the factor encoder, and performing nonlinear encoding through a multilayer perceptron containing two fully connected layers. Output a one-dimensional vector, expand this one-dimensional vector along the spatial dimension to match the dimension of the global image feature vector through a broadcast mechanism, and perform element-wise addition with the image feature vector to generate a joint cue embedding representation.

6. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step two, the multi-label classification supervision branch includes two parallel fully connected layers, which are used to predict the band type and degradation type of the image, respectively. During the training phase, the cross-entropy loss function and the multi-label binary cross-entropy loss function are used for supervision.

7. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step three, the specific implementation process of the affine transformation structure controlled by prompts is as follows: the joint prompt embedding representation is processed by a parameter generation module containing two parallel fully connected layers to generate the scale and shift required for the affine transformation, respectively. The l-th layer in the Transformer module of the image complex atom network and super-resolution backbone network is represented as follows: ； in, For the first Intermediate features of the layer The intermediate features of layer l are updated after embedding the affine transformation structure as follows: ； in, This represents element-wise multiplication. and These represent the ratio and bias parameters of the l-th layer, respectively.

8. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step three, the super-resolution backbone network is built based on SwinIR, and the original attention module is replaced with a gated fusion structure of compressed attention and neighborhood attention; The compressed attention mechanism is as follows: the key / value features are divided into blocks within a local window and compressed and mapped to obtain a low-dimensional compressed representation through depthwise separable convolution. The query features maintain full resolution and are subject to attention calculation. The neighborhood attention mechanism is as follows: each query feature is only subject to attention calculation with the key / value features in its k×k neighborhood. The gating fusion mechanism is as follows: the query features are input into the multilayer perceptron to generate gating weights of dimension 2. After being normalized by Softmax, these weights are applied to the compressed attention and neighborhood attention outputs respectively, and then fused into the final attention response.

9. The method for super-resolution of multi-band infrared images at arbitrary magnification according to claim 1, characterized in that, In step three, the image complex atom network and the super-resolution backbone network are pre-trained first, and then the network connections are jointly trained. Specifically, the image complex atom network and the super-resolution backbone network are connected in series to form a joint network. The output of the image complex atom network is not processed additionally and is directly used as the input of the super-resolution backbone network. The image encoder part retains the parameters obtained from the training of the image complex atom network, and the factor encoder part retains the parameters of the super-resolution backbone network. In the joint training stage, the dataset constructed in step 1.4 is used, and L1 loss is used for supervised optimization. The optimizer and learning rate strategy are consistent with those in the pre-training stage of the super-resolution backbone network.

10. A multi-band infrared image arbitrary magnification super-resolution system based on the method of any one of claims 1-9, characterized in that, The system includes: The multi-band infrared image data acquisition and construction module is used to acquire high-definition infrared images of different bands, construct training sets, validation sets and test sets containing different degradation types and low-resolution images at multiple magnifications, and record the band type label, degradation type label and scaling factor information of the corresponding images. The embedded generation module includes an image encoder and a factor encoder, which are used to extract global semantic features and scaling factor features of the image, and output the scaling and bias values ​​through feature fusion and affine transformation structure to achieve cross-task adaptive control. The super-resolution and restoration joint reconstruction network module consists of a joint network composed of an image complex atom network and a super-resolution backbone network. The image complex atom network is based on the Restormer structure and is responsible for handling multiple types of degradation. The super-resolution backbone network is based on the SwinIR structure and performs detail enhancement and resolution improvement. Both sub-networks embed affine transformation structures controlled by cueing, which are used to perform feature adaptive transformation according to image semantics and scaling requirements. The inference and output module receives infrared images of arbitrary resolution and a given magnification factor, performs frame-by-frame or batch processing through a joint network, and outputs high-resolution images with clear texture and consistent magnification.

Images