A multi-modal infrared image super-resolution reconstruction method, system, device and medium

Abstract

This application relates to a method, system, device, and medium for super-resolution reconstruction of multimodal infrared images. The method includes: acquiring a low-resolution thermal infrared image and a corresponding high-resolution visible light image; performing spatial registration processing on the high-resolution visible light image to obtain a registered visible light image; performing feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature maps and visible light feature maps; performing feature-level frequency domain fusion on the thermal infrared feature maps and visible light feature maps to generate a fused feature map; performing convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and then performing physical consistency high-frequency enhancement and physical consistency edge refinement processing on the initial super-resolution thermal infrared image to generate the final super-resolution thermal infrared image. This method effectively overcomes problems such as detail blurring, artifact generation, or physical property distortion existing in traditional methods.

Description

Technical Field

[0001] This invention belongs to the field of image super-resolution reconstruction technology, and in particular relates to a method, system, device and medium for multimodal infrared image super-resolution reconstruction. Background Technology

[0002] With the widespread application of infrared imaging technology in security monitoring, industrial inspection, and medical diagnosis, increasingly higher demands are being placed on the spatial resolution of infrared images. However, limited by the materials, processes, and physical diffraction limits of infrared detectors, acquiring high spatial resolution infrared images is typically costly and technically complex. Therefore, using algorithms for super-resolution reconstruction of low-resolution infrared images has become an economical and effective technical approach. In recent years, with the development of multimodal sensing technology, the idea of ​​fusing high-resolution visible light images to help improve the resolution of infrared images has attracted widespread attention, leading to the research direction of multimodal image super-resolution reconstruction.

[0003] Traditional methods for super-resolution reconstruction of multimodal images mainly focus on direct information fusion in the spatial domain. For example, some methods attempt to perform simple channel concatenation between a registered visible light image and an upsampled infrared image, and then input the concatenated image into a deep convolutional neural network for end-to-end reconstruction; other methods employ attention-based fusion strategies, aiming to adaptively select texture details in the visible light image in the spatial domain to enhance the infrared image.

[0004] However, these methods, which inherently rely on networks learning the correlations between modes within spatial features, still suffer from some inherent problems: infrared images and visible light images originate from different physical imaging mechanisms; the former reflects the thermal radiation characteristics of objects, while the latter reflects their reflective characteristics, and the two differ fundamentally in grayscale distribution and edge characteristics. Directly performing feature fusion or pixel-level operations in the spatial domain can easily introduce artifacts due to the mismatch in physical properties between modes, or cause key thermal information in infrared images to be overwhelmed by the strong texture details of visible light, resulting in information misalignment. More critically, existing methods lack constraints on the physical consistency of the fusion process, potentially disrupting the original thermal radiation distribution patterns of infrared images in pursuit of visual clarity, such as altering the relative temperature difference in local areas or the overall energy distribution. This is fatal for quantitative analysis applications that rely on absolute or relative temperature values ​​(such as fault diagnosis and body temperature screening). Summary of the Invention

[0005] Therefore, it is necessary to provide a super-resolution reconstruction method that can effectively introduce high-frequency details of visible light images while strictly preserving the physical properties of infrared images, in order to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a multimodal infrared image super-resolution reconstruction method, including:

[0007] Acquire a low-resolution thermal infrared image and its corresponding high-resolution visible light image; and use the low-resolution thermal infrared image as a reference to perform spatial registration processing on the high-resolution visible light image to obtain a registered visible light image.

[0008] Feature extraction processing was performed on the low-resolution thermal infrared image and the registered visible light image to obtain thermal infrared feature map and visible light feature map, respectively.

[0009] The thermal infrared feature map and the visible light feature map are fused in the frequency domain at the feature level to generate a fused feature map.

[0010] The fused feature map is convolved to generate an initial super-resolution thermal infrared image; and the initial super-resolution thermal infrared image is subjected to physical consistency high-frequency enhancement processing to obtain a high-frequency enhanced image.

[0011] Physical consistency edge thinning is performed on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0012] Furthermore, feature-level frequency domain fusion processing is performed on the thermal infrared feature map and the visible light feature map to generate a fused feature map, including:

[0013] Fast Fourier transforms were performed on the thermal infrared feature map and the visible light feature map respectively to obtain the thermal infrared frequency domain features and the visible light frequency domain features;

[0014] Amplitude difference gating weights are calculated based on the amplitude spectra of thermal infrared frequency domain characteristics and the amplitude spectra of visible light frequency domain characteristics.

[0015] Phase consistency gating weights are calculated based on the phase spectrum of thermal infrared frequency domain characteristics and the phase spectrum of visible light frequency domain characteristics.

[0016] The fusion weights of visible light features are calculated based on the preset high-frequency mask, amplitude difference gating weights, and phase consistency gating weights; the preset high-frequency mask is used to distinguish between high-frequency and low-frequency regions in the frequency domain.

[0017] Based on the fusion weights of visible light features, the visible light frequency domain features and the thermal infrared frequency domain features are weighted and fused to obtain the fused frequency domain features;

[0018] The fused frequency domain features are subjected to inverse fast Fourier transform to generate a fused feature map.

[0019] Furthermore, based on the amplitude spectrum of thermal infrared frequency domain characteristics and the amplitude spectrum of visible light frequency domain characteristics, the amplitude difference gating weights are calculated, including:

[0020] Calculate the absolute difference between the amplitude spectrum of the thermal infrared frequency domain characteristics and the amplitude spectrum of the visible light frequency domain characteristics;

[0021] The absolute difference values ​​are normalized to obtain the normalized magnitude difference.

[0022] Based on the normalized magnitude difference, the first initial gate value is calculated using the first learnable gate function:

[0023]

[0024] in, The first initial gate value, To normalize the difference in magnitude, and These are learnable parameters;

[0025] Based on the preset first calibration parameters, the first initial gate value is subjected to range calibration processing to obtain the functional amplitude difference gate weight; the first calibration parameters include amplitude scaling factor and amplitude offset.

[0026] Furthermore, based on the phase spectrum of thermal infrared frequency domain characteristics and the phase spectrum of visible light frequency domain characteristics, the phase consistency gating weights are calculated, including:

[0027] Calculate the absolute phase difference between the phase spectrum of the thermal infrared frequency domain characteristics and the phase spectrum of the visible light frequency domain characteristics;

[0028] The phase consistency index is obtained by performing complementation on the absolute phase difference value;

[0029] Based on the phase consistency index, the second initial gate value is calculated using a second learnable gate function:

[0030]

[0031] in, This is the second initial gate value. This is the absolute phase difference value. and These are learnable parameters;

[0032] Based on the preset second calibration parameters, the second initial gate value is subjected to range calibration processing to obtain the phase consistency gate weight; the second calibration parameters include the phase scaling coefficient and the phase offset.

[0033] Furthermore, based on the preset high-frequency mask, amplitude difference gating weight, and phase consistency gating weight, the fusion weights of the visible light features are calculated, including:

[0034] A high-frequency mask is generated based on the frequency coordinates; and high-frequency and low-frequency regions are determined based on the high-frequency mask; the high-frequency region is the area of ​​the high-frequency mask that is more than a certain distance from the frequency center than the cutoff frequency; the cutoff frequency is a learnable parameter.

[0035] Based on amplitude difference gating weights, phase consistency gating weights, and fundamental weights, the fusion weights for the high-frequency region are calculated using the following formula:

[0036]

[0037] in, Here, P represents the fusion weight for the high-frequency region, and P represents the phase consistency gating weight. The threshold weights are the amplitude difference gating weights, where clamp is the constraint function, 0.05 is the base weight, (0.05, 0, 999) is the weight value limit range, and 0.95 is the upper limit of the adjustment coefficient.

[0038] Set the fusion weight of the low-frequency region to 0; and merge the fusion weights of the high-frequency region and the low-frequency region to obtain the fusion weight of the visible light features.

[0039] Furthermore, the initial super-resolution thermal infrared image is subjected to physical consistency high-frequency enhancement processing to obtain a high-frequency enhanced image, including:

[0040] The registered visible light image is converted into a grayscale image; and the initial super-resolution thermal infrared image and grayscale image are processed by fast Fourier transform to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum and visible light amplitude spectrum.

[0041] The visible light amplitude spectrum is globally normalized to obtain the normalized visible light amplitude spectrum;

[0042] Based on the normalized visible light amplitude spectrum, thermal infrared amplitude spectrum, and a preset high-frequency mask, the enhanced amplitude spectrum in the high-frequency region is calculated using the following formula:

[0043]

[0044] in, The enhanced amplitude spectrum in the high-frequency region, The thermal infrared amplitude spectrum, For learnable enhancement strength coefficients, For the preset high-frequency mask, Normalized visible light amplitude spectrum;

[0045] The thermal infrared amplitude spectrum is determined as the enhanced amplitude spectrum in the low-frequency region; and the enhanced amplitude spectra in the high-frequency region and the enhanced amplitude spectra in the low-frequency region are combined to obtain the enhanced amplitude spectrum.

[0046] An enhanced image is generated by performing an inverse fast Fourier transform on the enhanced amplitude spectrum and the thermal infrared phase spectrum.

[0047] Based on the first attenuable mixing coefficient, the enhanced image is mixed with the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0048] Furthermore, the high-frequency enhanced image undergoes physical consistency edge refinement processing to generate a super-resolution thermal infrared image, including:

[0049] Extracting edge features and edge confidence masks from registered visible light images;

[0050] The features of the high-frequency enhanced image are stitched together with the edge features to generate the original residual image;

[0051] Based on physical constraints, the original residual plot is modified to obtain a constrained residual plot; the physical constraints include zero mean constraint and amplitude limit constraint.

[0052] The constraint residual map is weighted by using an edge confidence mask to obtain the final residual map;

[0053] Based on the second attenuable mixing coefficient, the final residual map and the high-frequency enhanced image are mixed to generate a super-resolution thermal infrared image.

[0054] Secondly, this application also provides a multimodal infrared image super-resolution reconstruction system, comprising:

[0055] The image registration module is used to acquire a low-resolution thermal infrared image and the corresponding high-resolution visible light image; and to perform spatial registration processing on the high-resolution visible light image based on the low-resolution thermal infrared image to obtain a registered visible light image.

[0056] The feature extraction module is used to perform feature extraction processing on low-resolution thermal infrared images and registered visible light images respectively, to obtain thermal infrared feature maps and visible light feature maps;

[0057] The feature fusion module is used to perform feature-level frequency domain fusion processing on thermal infrared feature maps and visible light feature maps to generate fused feature maps.

[0058] The image enhancement module is used to perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and to perform physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0059] The super-resolution image generation module is used to perform physical consistency edge refinement on high-frequency enhanced images to generate super-resolution thermal infrared images.

[0060] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement any of the multimodal infrared image super-resolution reconstruction methods described in the embodiments of this application.

[0061] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement a multimodal infrared image super-resolution reconstruction method as described in any of the embodiments of this application.

[0062] The aforementioned method, system, device, and medium for super-resolution reconstruction of multimodal infrared images acquire and accurately register multimodal images, extract features from the registered images, perform feature-level frequency domain fusion to generate a fused feature map, and then perform physical consistency high-frequency enhancement and physical consistency edge refinement on the fused feature map to obtain a super-resolution thermal infrared image. The entire process utilizes innovative gated fusion and relative enhancement mechanisms to adaptively introduce visible light details, while strict phase preservation and physical constraints ensure that the reconstruction result does not deviate from the physical properties of the thermal infrared image (such as heat distribution patterns). This effectively overcomes problems such as detail blurring, artifact generation, or distortion of physical properties inherent in traditional methods. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating a multimodal infrared image super-resolution reconstruction method in one embodiment;

[0065] Figure 2 This is a schematic diagram of the steps in one embodiment to perform feature-level frequency domain fusion processing on thermal infrared feature maps and visible light feature maps to generate fused feature maps;

[0066] Figure 3 This is a schematic diagram of the structure of a multimodal infrared image super-resolution reconstruction system in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In one embodiment, a multimodal infrared image super-resolution reconstruction method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0069] Step S101: Acquire a low-resolution thermal infrared image and the corresponding high-resolution visible light image; and use the low-resolution thermal infrared image as a reference to perform spatial registration processing on the high-resolution visible light image to obtain a registered visible light image.

[0070] Spatial registration refers to unifying spatial data such as images and point clouds from different sources, times, resolutions, or coordinate systems into a single standard spatial reference frame, eliminating positional discrepancies between data, and ensuring that they correspond precisely in spatial location.

[0071] For example, when acquiring a low-resolution thermal infrared image and a corresponding high-resolution visible light image of the same scene using an image acquisition device, spatial positional deviations exist between the two directly acquired images due to differences in physical location and imaging characteristics between the thermal infrared sensor and the visible light sensor. A feature-point-based registration method can be employed, using the low-resolution thermal infrared image as a spatial reference to perform precise spatial registration processing on the high-resolution visible light image. This involves calculating a geometric transformation model (such as an affine transformation) from the visible light image to the thermal infrared image, and then using this model to resample the visible light image. This ensures that each pixel in the visible light image is precisely aligned spatially with its corresponding scene point in the thermal infrared image, resulting in a registered visible light image. The feature-point-based registration method achieves the alignment of two or more images through three steps: extracting unique image features, matching feature correspondences, and calculating the spatial transformation model. It is commonly used in image stitching, target tracking, and 3D reconstruction. Resampling refers to adjusting the sampling frequency or sample density of the data, i.e., increasing (upsampling) or decreasing (downsampling) the number of data points according to specific rules to adapt to subsequent analysis, storage, or application requirements.

[0072] Step S102: Perform feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature map and visible light feature map.

[0073] Feature extraction refers to the core steps of filtering, transforming, and extracting key information (i.e., features) that are meaningful to the target task from raw data such as images and text. The purpose is to transform the pixel information of the original image into a high-dimensional feature representation that can better represent the content of the image.

[0074] For example, after obtaining a spatially aligned registered visible light image, a convolutional neural network can be used to extract features from both the low-resolution thermal infrared image and the registered visible light image. For the low-resolution thermal infrared image, it can first be upsampled (e.g., bicubic interpolation) to the target resolution, and then its thermal infrared feature map can be extracted using a lightweight convolutional encoder. This feature map mainly contains the target's thermal distribution contour and thermal contrast information. For the registered visible light image, a more complex feature extraction network (e.g., containing residual blocks or densely connected blocks) can be used to fully extract its rich texture details and edge information to obtain the visible light feature map. The two feature extraction networks can share some structures or be designed independently, but it is necessary to ensure that the output thermal infrared feature map and visible light feature map are consistent in spatial dimensions. The convolutional neural network (CNN) is used in this process. Network (CNN) is a deep learning model inspired by biological visual systems. It can efficiently extract spatial features. Its core design revolves around reducing the number of parameters and preserving spatial correlations, avoiding the problems of parameter explosion and loss of spatial information when traditional neural networks process images. Upsampling is a core technology in digital image processing to improve image resolution. Simply put, it uses algorithms to "supplement" pixels, turning a low-resolution image (few pixels) into a high-resolution image (more pixels). Bicubic interpolation refers to the process of assigning different values ​​to the 16 neighboring pixels around a reference target pixel (an extended 4x4 region around a 2x2 pixel block in the original image). Pixels are weighted, with larger weights (stronger influence) for pixels closer to the target pixel and smaller weights for pixels farther away. The weights are calculated using a cubic function (such as a cubic convolution function) to ensure a smooth transition. The values ​​of the 16 pixels are then multiplied by their corresponding weights and summed to obtain the final value of the newly added pixel. The lightweight convolutional encoder is an efficient encoding structure optimized through parameter compression and computational simplification based on the traditional convolutional encoder. Its goal is to significantly reduce the number of model parameters, computational cost, and memory usage while maintaining certain encoding performance (such as feature extraction). The feature extraction network is the core module in deep learning responsible for extracting valuable information (such as features) from raw data (such as image pixels).

[0075] Step S103: Perform feature-level frequency domain fusion processing on the thermal infrared feature map and the visible light feature map to generate a fused feature map.

[0076] Feature-level frequency domain fusion is a technical approach in the fusion of cross-modal data such as text and images. It integrates the features of different modal data in the frequency domain space, rather than directly fusing them in the original time / spatial domain (such as pixels) or decision layer (such as the final classification result).

[0077] For example, feature-level frequency domain fusion processing is performed on the thermal infrared feature map and the visible light feature map to generate a fused feature map that retains both the characteristics of thermal infrared and the detailed texture of visible light.

[0078] Step S104: Perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and perform physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0079] Convolution processing refers to the use of a "kernel" (i.e., a small function / matrix) to perform a sliding weighted summation with the original signal (function / data) to extract or transform features of the original signal. Physically consistent high-frequency enhancement processing is a technique commonly used in signal processing or data optimization fields such as audio and images. It enhances the high-frequency components in the signal / data while strictly adhering to the physical laws or real attributes of the signal itself, avoiding distortion or violation of real-world logic due to enhancement. High-frequency components correspond to information such as details and sharpness, such as the edges of an image and the overtones of audio.

[0080] For example, the generated fused feature map is passed through a convolutional layer for channel number adjustment and feature integration to generate an initial super-resolution thermal infrared image, which already possesses higher clarity than the original low-resolution input. To further improve image quality and ensure physical plausibility, the initial super-resolution thermal infrared image is then subjected to physically consistent high-frequency enhancement processing to obtain a high-frequency enhanced image. The convolutional layer is typically a 1x1 convolution.

[0081] Step S105: Perform physical consistency edge refinement processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0082] Physical consistency edge refinement is a technique in the field of computer vision and image processing that optimizes the edge details of an image while ensuring that the result conforms to the physical laws of the real world and avoids edge distortion that goes against common sense.

[0083] For example, a high-frequency enhanced image is subjected to physical consistency edge refinement, with a focus on optimizing details, to generate a super-resolution thermal infrared image.

[0084] In this embodiment, multimodal images are acquired and precisely registered. Feature extraction is then performed on the registered images, followed by feature-level frequency domain fusion to generate a fused feature map. This fused feature map undergoes physical consistency high-frequency enhancement and physical consistency edge refinement processing. This successfully and effectively transfers rich detail information from high-resolution visible light images to the reconstruction process of low-resolution thermal infrared images, ultimately resulting in a super-resolution thermal infrared image. The entire process utilizes innovative gated fusion and relative enhancement mechanisms to adaptively introduce visible light details. Simultaneously, strict phase preservation and physical constraints ensure that the reconstruction result does not deviate from the physical properties of the thermal infrared image (such as thermal distribution patterns). This effectively overcomes problems such as detail blurring, artifact generation, or physical property distortion inherent in traditional methods.

[0085] In one embodiment, such as Figure 2 As shown, feature-level frequency domain fusion processing is performed on the thermal infrared feature map and the visible light feature map to generate a fused feature map, including:

[0086] Step S201: Perform fast Fourier transform on the thermal infrared feature map and the visible light feature map respectively to obtain thermal infrared frequency domain features and visible light frequency domain features.

[0087] The Fast Fourier Transform (FFT) is an efficient computational algorithm for the Discrete Fourier Transform (DFT), which significantly reduces the computational complexity of the DFT by using the divide-and-conquer approach. The DFT is a mathematical tool for converting discrete-time domain signals into discrete-frequency domain signals.

[0088] For example, thermal infrared and visible light feature maps are treated as two-dimensional discrete signals and transformed from the spatial domain to the frequency domain using the FFT algorithm, yielding corresponding complex-form frequency domain features, namely thermal infrared frequency domain features and visible light frequency domain features. Both thermal infrared and visible light frequency domain features consist of two components: an amplitude spectrum and a phase spectrum. The amplitude spectrum characterizes the energy intensity of each frequency component and reflects the contrast information of the feature; the phase spectrum records the relative positional relationships of the frequency components and carries the structural topological information of the feature. During implementation, the integrity of the transformation must be ensured, for example, by maintaining spectral resolution through zero-padding and standardizing the storage format of the complex results. Zero-padding is a basic operation commonly used in data processing and signal analysis, where zero values ​​are added to the original data to adjust the data length, dimension, or to meet the requirements of specific algorithms.

[0089] Step S202: Calculate the amplitude difference gating weight based on the amplitude spectrum of thermal infrared frequency domain characteristics and the amplitude spectrum of visible light frequency domain characteristics.

[0090] Among them, the amplitude spectrum, as a quantitative representation of the frequency energy distribution, directly reflects the degree of deviation between the thermal infrared and visible light modes in terms of characteristic intensity.

[0091] For example, the absolute difference between the amplitude spectrum of the thermal infrared frequency domain features and the amplitude spectrum of the visible light frequency domain features is calculated. Based on this difference, an amplitude difference gating weight is calculated using a learnable gating function. The learnable gating function is designed parametrically (e.g., by introducing parameterizable parameters). and Dynamically adjust the weight response curve.

[0092] Step S203: Calculate the phase consistency gating weights based on the phase spectrum of thermal infrared frequency domain characteristics and the phase spectrum of visible light frequency domain characteristics.

[0093] Among them, the consistency of the phase spectrum, as the structural carrier of frequency domain features, is the key to ensuring multimodal spatial alignment.

[0094] For example, the absolute phase difference between the thermal infrared and visible light phase spectra is calculated. This phase difference is then mathematically transformed into a consistency index, ensuring a positive correlation between the index value and consistency. Based on the processed index, a learnable gating function is used to calculate the phase consistency gating weight. The absolute phase difference reflects the degree of frequency offset in the spatial structure; the learnable gating function is designed parametrically (e.g., by introducing parameters). and Dynamically adjust the sensitivity to phase deviation.

[0095] Step S204: Calculate the fusion weight of visible light features based on the preset high-frequency mask, amplitude difference gating weight, and phase consistency gating weight; the preset high-frequency mask is used to distinguish the high-frequency region and the low-frequency region in the frequency domain.

[0096] The preset high-frequency mask is a binary mask generated based on frequency coordinates, used to divide the high-frequency and low-frequency regions in the frequency domain. The high-frequency region corresponds to the detailed components of the image (such as edge texture), and the low-frequency region corresponds to the macroscopic structure.

[0097] For example, the processing area is distinguished according to the preset high-frequency mask, and the visible light feature fusion weight is calculated based on the determined processing area, amplitude difference gating weight and phase consistency gating weight.

[0098] Step S205: Based on the fusion weight of visible light features, the visible light frequency domain features and thermal infrared frequency domain features are weighted and fused to obtain fused frequency domain features.

[0099] Among them, weighted fusion processing refers to integrating the advantageous components of multimodal features into a unified expression through linear combination.

[0100] For example, for each spatial frequency location, the visible light frequency domain features and the thermal infrared frequency domain features are weighted and summed according to the fusion weights. The weighting formula is as follows: Here, w represents the fusion weight of the visible light features. Positions with high weight values ​​emphasize the contribution of visible light features to incorporate detailed information, while positions with low weight values ​​retain the original content of the thermal infrared features, maintaining their physical properties. It's important to note that since the frequency domain features are complex numbers, fusion requires handling complex number operations, either by weighting the real and imaginary parts separately, or by performing linear operations directly on the complex number as a whole. During the fusion process, it's crucial to ensure that the weight map and feature map are sized correctly. The resulting fused frequency domain feature contains both the texture details of visible light and retains the thermal radiation characteristics of thermal infrared, forming a complementary and unified representation.

[0101] Step S206: Perform inverse fast Fourier transform on the fused frequency domain features to generate a fused feature map.

[0102] The Inverse Fast Fourier Transform (IFFT) is the inverse operation of the Fast Fourier Transform (FFT) and is used to convert frequency domain signals back to time domain signals.

[0103] For example, applying the inverse fast Fourier transform to the fused frequency domain features converts the complex-form frequency domain data into a real-valued spatial feature map, i.e., the fused feature map. This fused feature map simultaneously contains multiple modes, primarily referring to visible light and thermal infrared data: high-frequency components are enhanced by visible light features, exhibiting clearer edge textures; low-frequency components continue the thermal distribution contours of thermal infrared features. During this process, it is crucial to maintain spectral conjugate symmetry to ensure the real-valued nature of the reconstruction result.

[0104] In this embodiment, the extracted thermal infrared and visible light feature maps are transformed from spatial domain features to frequency domain features. An amplitude and phase gating mechanism is used to adaptively calculate the fusion weights, and after weighted fusion in the frequency domain, the images are inversely transformed back to the spatial domain to obtain the fused feature map. This effectively avoids the modal conflict problem commonly encountered in spatial domain fusion, significantly improves detail representation, and maintains the physical essence of the thermal infrared image.

[0105] In one embodiment, the amplitude difference gating weight is calculated based on the amplitude spectrum of thermal infrared frequency domain characteristics and the amplitude spectrum of visible light frequency domain characteristics, including:

[0106] Step S301: Calculate the absolute difference between the amplitude spectrum of the thermal infrared frequency domain feature and the amplitude spectrum of the visible light frequency domain feature.

[0107] For example, amplitude spectrum components are extracted from the thermal infrared and visible light frequency domain features obtained through Fast Fourier Transform (FFT). Then, pointwise absolute difference calculations are used to obtain the absolute difference values ​​between the amplitude spectra: for each corresponding coordinate position in the frequency domain, the thermal infrared amplitude spectrum value is subtracted from the visible light amplitude spectrum value, and the absolute value is taken to generate an absolute difference map with the same size as the original feature map. This difference map is used to visually characterize the inconsistency in energy distribution of the two modes at different frequency components: regions with smaller difference values ​​indicate that the frequency component has similar intensity in both modes, possibly corresponding to shared structural information; while regions with larger difference values ​​suggest the possible presence of mode-specific noise or texture imaging artifacts. The pointwise absolute difference calculation is a basic element-level mathematical operation, which can be broken down into two steps: pointwise processing and absolute difference calculation. For two datasets with identical dimensions (such as an image pixel array), the difference between elements is calculated one by one at corresponding positions, and then the absolute value of the difference is taken, ultimately obtaining an absolute difference result set with the same dimensions as the original dataset.

[0108] Step S302: Normalize the absolute difference values ​​to obtain the normalized magnitude difference.

[0109] In this case, the absolute difference value is not normalized. The difference in the overall brightness or contrast of the feature map will directly dominate the magnitude of the absolute difference value, thus masking the important local relative difference patterns. Normalization refers to mapping the original data of different magnitudes and units to a unified numerical range, usually [0,1] or [-1,1], to eliminate the influence of the overall energy scale of the feature and make the difference measurement comparable.

[0110] For example, the absolute difference value can be divided by a scaling factor, which can be the sum of the corresponding position values ​​of the thermal infrared amplitude spectrum and the visible light amplitude spectrum plus a very small constant for numerical stability, i.e., the relative difference ratio is calculated, and finally the normalized amplitude difference is obtained.

[0111] Step S303: Based on the normalized amplitude difference, calculate the first initial gate value using the first learnable gate function.

[0112]

[0113] in, The first initial gate value, To normalize the difference in magnitude, and These are learnable parameters.

[0114] The first learnable gating function is the hyperbolic tangent function (tanh), which has S-shaped curve characteristics and can smoothly compress the input into the interval (-1,1). Used to control the steepness of the function curve's response, i.e., the sensitivity of the gate value to changes in difference; It acts as a threshold or offset to determine the center point of the function, that is, the level of input difference corresponding to an output of 0.5; and These two parameters are optimized during training using the backpropagation algorithm, enabling the gating mechanism to adaptively learn what magnitude of amplitude difference should lead to what degree of gating inhibition or enhancement.

[0115] For example, the normalized magnitude difference is substituted into the formula, first a linear transformation and shift are performed, then compression is achieved using the tanh function, and finally the output range is adjusted to the (0,1) interval through a linear transformation to obtain the first initial gate value. The closer the initial gate value is to 1, the smaller the difference at that position, tending to preserve or enhance visible light features; the closer it is to 0, the larger the difference, tending to suppress visible light features. The backpropagation algorithm adjusts all parameters in the network (such as weights and biases) by calculating the error in reverse, ultimately making the neural network's prediction result closer to the true label. Essentially, it is an efficient implementation of gradient descent optimization in multi-layer networks.

[0116] Step S304: Based on the preset first calibration parameters, the first initial gate value is subjected to range calibration processing to obtain the functional amplitude difference gate weight; the first calibration parameters include amplitude scaling factor and amplitude offset.

[0117] Among them, range calibration is a technical operation that corrects errors and unifies the scale of the output range by referencing a benchmark, in order to ensure the accuracy, consistency and reliability of the data; scaling factor is used to stretch or compress the distribution range of the first initial gate value to enhance its discriminativeness; offset is used to shift the center of the first initial gate value as a whole to adapt to the expected input range of the subsequent fusion formula.

[0118] For example, although the first initial gate value is within the range of (0,1), its numerical distribution may not yet reach the optimal dynamic range or center position required for the fusion weights. A linear transformation can be applied to the first initial gate value, and the transformation formula is as follows: Where scalemag is a preset or learnable amplitude scaling factor, and offsetmag is a preset or learnable amplitude offset. The calibrated result is the final amplitude difference gating weight. To ensure the reasonableness of the weights, a constraint function is usually applied to strictly limit their values ​​to a certain effective range, such as [0.05, 0.95], to prevent extreme weights from affecting training stability. The final amplitude difference gating weight is a weight map of the same size as the input feature map, where each pixel value represents the credibility or contribution of the visible light feature at the corresponding spatial frequency location based on amplitude spectrum similarity.

[0119] In this embodiment, the absolute difference between the amplitude spectra of thermal infrared and visible light frequencies is calculated and normalized. Then, a learnable gating function is used to map the relative difference into an initial gating response. After linear calibration, accurate weights suitable for feature fusion are obtained. This significantly reduces the need for manually setting fixed fusion rules, improves the robustness of the model in different scenarios, and provides key weight control signals for achieving high-quality, physically consistent multimodal image fusion.

[0120] In one embodiment, phase consistency gating weights are calculated based on the phase spectrum of thermal infrared frequency domain characteristics and the phase spectrum of visible light frequency domain characteristics, including:

[0121] Step S401: Calculate the absolute phase difference between the phase spectrum of the thermal infrared frequency domain feature and the phase spectrum of the visible light frequency domain feature.

[0122] The phase spectrum records the relative positional relationship of each frequency component in the spatial domain and is the carrier of the image structure.

[0123] For example, phase spectrum components are extracted from the thermal infrared and visible light frequency domain features obtained through Fast Fourier Transform (FFT). Since phase values ​​are periodic angular quantities, calculating the absolute phase difference requires special handling of the angular wrapping problem. A point-by-point calculation method can be used, performing difference operations on corresponding positions in the two phase spectra, and constraining the results within a reasonable angular range through modulo operations to ensure the geometric meaning of the difference is clear. The resulting absolute phase difference map can intuitively reflect the alignment errors of the two spatial structures: regions with differences close to zero indicate that the structure corresponding to that frequency component is highly consistent in position in both modes; while regions with larger differences suggest possible registration residuals, motion blur, or mode-specific structural expression differences. Point-by-point calculation is a basic numerical calculation logic that breaks down the overall calculation target into multiple independent points, i.e., the smallest calculation units. Then, the same or corresponding calculation rules are performed on each point one by one, and finally the results of all points are integrated to complete the overall task. The modulo operation, usually represented by the symbol % or the remainder, is to calculate the remainder after dividing two integers. The sign of the remainder is usually the same as that of the dividend. The rules vary slightly in different programming languages, but the core logic is the same.

[0124] Step S402: Perform complementation on the absolute phase difference to obtain the phase consistency index.

[0125] Among them, the complement processing is to replace the subtraction of the original number with the addition of the complement, which transforms subtraction into addition, which is easier to implement in hardware.

[0126] For example, the absolute phase difference value can be converted into a consistency index through mathematical transformation, for example, using the formula " "Alternatively, the difference can be normalized and then inverted. This transformation makes the output value positively correlated with consistency: when the absolute phase is small, the consistency index approaches its maximum value; when the difference increases, the index value decreases. During implementation, attention must be paid to numerical stability to avoid abnormal situations such as division by zero. The phase consistency index obtained after the transformation has a clear physical meaning and is used to characterize the degree of consistency of the spatial structure of the corresponding frequency components in the thermal infrared and visible light modes."

[0127] Step S403: Based on the phase consistency index, calculate the second initial gate value using the second learnable gate function.

[0128]

[0129] in, This is the second initial gate value. This is the absolute phase difference value. and These are learnable parameters.

[0130] in, The slope of the control function curve determines the sensitivity of the gate value to changes in consistency. As a threshold parameter, it defines the minimum standard for consistency requirements; the second learnable gating function is based on the hyperbolic tangent function. and These two parameters are optimized during training using the backpropagation algorithm, enabling the gating mechanism to adaptively learn what degree of structural alignment is reliable.

[0131] For example, by substituting the consistency index into the formula and performing linear transformation, hyperbolic tangent compression, and output range adjustment, a second initial gate value P distributed in the (0,1) interval is obtained. The larger the value of P, the higher the phase consistency at that position, and the more reliable the structural information of the visible light features.

[0132] Step S404: Based on the preset second calibration parameters, the second initial gate value is subjected to range calibration processing to obtain the phase consistency gate weight; the second calibration parameters include the phase scaling coefficient and the phase offset.

[0133] For example, based on a preset phase scaling factor and phase offset, the phase consistency gating weight is calculated using a linear transformation formula: ,in, For phase consistency gating weights, This is a preset phase scaling factor used to adjust the dynamic range of the gate value. A preset phase offset is used to shift the numerical center. To ensure the weights are reasonable, a range constraint operation is usually applied to strictly constrain the results within a valid range. The final phase consistency gated weight is a weight map of the same size as the input feature map. Each value in the map can be quantified at its corresponding spatial frequency position, representing the reliability of the visible light feature structure information based on phase consistency evaluation. The range constraint operation refers to applying rule constraints to the value range (range) of a data / variable to ensure that the final result falls within a preset valid interval.

[0134] In this embodiment, the absolute difference between the phase spectra in the thermal infrared and visible light frequency domains is calculated and converted into a consistency index through complementation. A parameterized, learnable gating function is then used to map the consistency level to a preliminary gating response, which is linearly calibrated to obtain precise weights suitable for feature fusion. This effectively avoids ghosting or artifacts caused by structural misalignment. This phase consistency-based adaptive gating mechanism effectively complements amplitude gating, jointly ensuring the accuracy and stability of structural information during multimodal fusion.

[0135] In one embodiment, the fusion weights of visible light features are calculated based on a preset high-frequency mask, amplitude difference gating weights, and phase consistency gating weights, including:

[0136] Step S501: Generate a high-frequency mask based on the frequency coordinates; and determine the high-frequency region and the low-frequency region based on the high-frequency mask; the high-frequency region is the area of ​​the high-frequency mask that is more than a certain distance from the frequency center than the cutoff frequency; the cutoff frequency is a learnable parameter.

[0137] The high-frequency mask is a binary matrix with the same size as the frequency domain feature map, where the positions corresponding to high-frequency regions are assigned a value of 1 and the positions corresponding to low-frequency regions are assigned a value of 0. The generation of this mask does not depend on specific input data, but automatically learns the optimal cutoff frequency during the training process, thereby adaptively determining the range of the most critical frequency band for reconstruction.

[0138] For example, a frequency coordinate system is constructed with the frequency domain center (i.e., the DC component) as the origin, and the frequency value of each point is determined by its radial distance from the origin. Based on this coordinate system, the boundary between high-frequency and low-frequency regions is defined by a learnable parameter cutoff frequency: the region beyond the frequency center is identified as the high-frequency region, while the region within it is classified as the low-frequency region.

[0139] Step S502: Based on the amplitude difference gating weight, phase consistency gating weight, and basic weight, the fusion weight of the high-frequency region is calculated using the following formula:

[0140]

[0141] in, Here, P represents the fusion weight for the high-frequency region, and P represents the phase consistency gating weight. The weights are used for amplitude difference gating, where clamp is the constraint function, 0.05 is the base weight, (0.05, 0, 999) is the weight value range, and 0.95 is the upper limit of the adjustment coefficient.

[0142] Here, 0.05 is used as the base weight to ensure that even when the two modal features differ greatly, the visible light features still maintain a minimum contribution; 0.95 is the upper limit of the adjustment coefficient, which, together with the base weight, constrains the theoretical upper limit of the total weight to 1.0; the clamp function strictly limits the final weight value to the range [0.05, 0.999] to prevent numerical instability during training.

[0143] For example, the obtained amplitude difference gating weights and phase consistency gating weights are fused according to the fusion formula to obtain the fusion weights for the high-frequency region.

[0144] Step S503: Set the fusion weight of the low-frequency region to 0; and merge the fusion weight of the high-frequency region and the fusion weight of the low-frequency region to obtain the fusion weight of the visible light features.

[0145] For example, based on the physical nature of thermal infrared images, for all locations identified as low-frequency regions by high-frequency masks, their fusion weights are directly set to 0. The physical nature of thermal infrared images is that the low-frequency components primarily carry the overall brightness, contrast, and large-scale heat; these are the most critical physical properties of thermal infrared images. Therefore, completely excluding the influence of visible light features in the low-frequency regions ensures that the reconstructed image maintains consistency with the original thermal infrared image in terms of overall thermal radiation distribution characteristics, avoiding potential temperature value shifts or contrast distortions caused by the introduction of visible light information. For example, through a simple matrix merging operation, the non-zero weights calculated in the high-frequency regions are merged with the zero weights in the low-frequency regions to generate a complete fusion weight map with the same size as the original frequency domain feature map. This weight map precisely indicates the contribution proportion of visible light feature information in the final fusion result at different locations in the frequency domain. Matrix merging refers to the basic operation of combining two or more structurally compatible matrices into a new matrix according to specific rules, ensuring dimensional consistency in the merging direction.

[0146] In this embodiment, a high-frequency mask is generated to adaptively divide the frequency domain. Intelligent weights for the high-frequency region are calculated by combining amplitude and phase consistency constraints. Based on the principle of physical consistency, the dominant position of thermal infrared features is fully preserved in the low-frequency region, ultimately constructing a fusion weight map covering the entire frequency domain. This effectively solves the contradiction between rich detail and physical realism in traditional methods, providing a crucial weight control blueprint for generating super-resolution infrared images that possess both high spatial resolution and strict thermal-physical consistency.

[0147] In one embodiment, a physically consistent high-frequency enhancement process is performed on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image, including:

[0148] Step S601: Convert the registered visible light image into a grayscale image; and perform fast Fourier transform processing on the initial super-resolution thermal infrared image and the grayscale image respectively to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum and visible light amplitude spectrum.

[0149] For example, a weighted summation method can be used to convert a registered visible light image from the RGB color space to a grayscale image. This involves assigning specific weight coefficients to the red, green, and blue channels and then linearly combining them, thus preserving the image's luminance information while eliminating chromaticity interference. Subsequently, a Fast Fourier Transform (FFT) is performed on both the initial super-resolution thermal infrared image and the generated grayscale image. This transform maps the spatial domain to the frequency domain, decomposing the image into sinusoidal components of different frequencies. The resulting thermal infrared frequency domain representation contains two independent components: an amplitude spectrum and a phase spectrum. The amplitude spectrum characterizes the energy distribution intensity of each frequency component, while the phase spectrum records the spatial structure relationship of the frequency components. The frequency domain representation of the visible light image also contains these two components. The weighted summation method involves assigning corresponding coefficients to different factors based on their importance (i.e., weights), multiplying the values ​​of each factor by their corresponding weights, and then summing the results to obtain the final comprehensive result.

[0150] Step S602: Perform global normalization on the visible light amplitude spectrum to obtain the normalized visible light amplitude spectrum.

[0151] Global normalization is based on global statistical information of the entire dataset. It scales or offsets the data to make the data conform to specific distribution characteristics. Its purpose is to eliminate the influence of overall image brightness differences on the enhancement process and to convert the amplitude value into an indicator of the degree of deviation relative to the global average level.

[0152] For example, the global arithmetic mean of the visible light amplitude spectrum is calculated, which represents the overall energy level of the image's frequency components. The value at each location in the amplitude spectrum is then divided by this mean to obtain the normalized visible light amplitude spectrum. After this processing, the mean of the new amplitude spectrum becomes 1, and its numerical distribution indicates that regions greater than 1 correspond to energy-prominent areas relative to the average level, typically richly detailed edge textures, while regions less than 1 correspond to smooth areas with energy below the average level.

[0153] Step S603: Based on the normalized visible light amplitude spectrum, thermal infrared amplitude spectrum, and a preset high-frequency mask, calculate the enhanced amplitude spectrum in the high-frequency region using the following formula:

[0154]

[0155] in, The enhanced amplitude spectrum in the high-frequency region, The thermal infrared amplitude spectrum, For learnable enhancement strength coefficients, For the preset high-frequency mask, This is the normalized visible light amplitude spectrum.

[0156] Among them, the learnable enhancement intensity coefficient α is used to control the overall magnitude of the enhancement.

[0157] For example, the aforementioned generated high-frequency mask is reused to identify high-frequency regions. Within these high-frequency regions, an enhancement formula is applied to calculate the enhanced amplitude spectrum. This formula uses the thermal infrared amplitude spectrum as a basis, and the enhancement amount is determined by the degree of deviation of the normalized visible light amplitude spectrum from the neutral value of 1 (i.e., ) decided, when When, it indicates that the visible light details at that location are significant, and positive enhancement is performed; when When the visible light is smooth at that point, it appropriately weakens the energy of the corresponding frequency in the thermal infrared, avoiding the introduction of noise; when When the original value of thermal infrared radiation remains unchanged, it is assumed that the value remains unchanged.

[0158] Step S604: Determine the thermal infrared amplitude spectrum as the enhanced amplitude spectrum in the low-frequency region; and merge the enhanced amplitude spectrum in the high-frequency region and the enhanced amplitude spectrum in the low-frequency region to obtain the enhanced amplitude spectrum.

[0159] For example, to ensure strict preservation of physical properties, a differentiated processing strategy is adopted for the low-frequency and high-frequency regions, and spectral synthesis is performed. Based on the high-frequency mask, the frequency domain is clearly divided into two regions: for the low-frequency region, based on important physical considerations, the thermal infrared amplitude spectrum is directly determined as the enhanced amplitude spectrum for that region without any modification. The important physical considerations refer to the fact that the low-frequency components of the image dominate the overall brightness distribution and macroscopic contrast. This information directly corresponds to the most critical absolute or relative temperature characterization in the thermal infrared image. Any modification to this information may introduce unacceptable temperature measurement errors; therefore, absolutely preserving low-frequency information is the cornerstone of maintaining the interpretability of thermal infrared images. For example, the unmodified amplitude spectrum of the low-frequency region is merged with the enhanced amplitude spectrum calculated from the high-frequency region to generate the final enhanced amplitude spectrum covering the entire frequency domain.

[0160] Step S605: Perform inverse fast Fourier transform on the enhanced amplitude spectrum and thermal infrared phase spectrum to generate an enhanced image.

[0161] For example, the original thermal infrared phase spectrum extracted from the initial super-resolution thermal infrared image is strictly combined with the calculated enhanced amplitude spectrum to form the final enhanced frequency domain representation. Maintaining the phase spectrum is another key measure to ensure physical consistency, as phase information determines the position of frequency components in the spatial domain, and any change will lead to geometric distortion of the image structure. Subsequently, an inverse fast Fourier transform is performed on this combined frequency domain representation to return the frequency domain to the spatial domain, generating the enhanced image. This image should visually exhibit sharper edges and richer textures, while its core thermal structure is perfectly aligned with the original thermal infrared image.

[0162] Step S606: Based on the first attenuable mixing coefficient, the enhanced image is mixed with the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0163] Among them, the first attenuable mixing coefficient As the number of training rounds gradually decreases, in the early stages of training, The value indicates that the output tends to retain the stable features of the initial image, which helps the network to initially converge. As training progresses, As the value linearly decays to zero, the proportion of the enhanced image gradually increases, allowing the physical consistency enhancement effect to take effect slowly and smoothly. This progressive hybrid strategy serves as an effective regularization method to avoid impacting the entire network training due to the initial unsatisfactory output of the enhancement module, ensuring the robustness of the optimization process. Enhancement modules (such as feature enhancement) are usually key auxiliary units of the network, whose function is to optimize the quality of core features and pass the optimized features to the subsequent backbone network (such as the classification head).

[0164] For example, through a first decaying mixing coefficient that gradually decreases with the number of training epochs. The newly generated enhanced image is then weighted and mixed with the original initial super-resolution thermal infrared image to obtain a high-frequency enhanced image. The mixing formula is as follows:

[0165]

[0166] in, For high-frequency enhanced images, The first attenuable mixing coefficient, To enhance the image, This is the initial super-resolution thermal infrared image.

[0167] In this embodiment, a fast Fourier transform is performed on the multimodal image to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum, and visible light amplitude spectrum. The visible light amplitude spectrum is normalized and converted into a relative intensity indicator. Then, an enhanced amplitude spectrum for the high-frequency region is generated based on a high-frequency mask and a relative enhancement formula, while the thermal infrared amplitude spectrum is used as the enhanced amplitude spectrum for the low-frequency region and the two spectra are merged to obtain an enhanced amplitude spectrum covering the entire frequency domain. The spatial domain image is reconstructed through an inverse transform. This approach achieves an organic unity between detail enhancement and physical property preservation, meeting the detail requirements of advanced vision tasks while ensuring the stringent physical realism requirements of quantitative thermal analysis applications.

[0168] In one embodiment, physical consistency edge refinement is performed on the high-frequency enhanced image to generate a super-resolution thermal infrared image, including:

[0169] Step S701: Extract edge features and edge confidence masks from the registered visible light image.

[0170] The edge confidence mask is a matrix of the same size as the feature map, with a value range between 0 and 1, used to represent the confidence level of the existence of an edge at the corresponding location.

[0171] For example, edge features can be extracted from the registered visible light image using classic edge detection operators to obtain an edge feature map. Simultaneously, an edge confidence mask can be computed in parallel, with the confidence level calculated based on the intensity of the edge features (e.g., through compression using the sigmoid function). High-confidence regions indicate that the edge structure is clear and well-defined in the visible light image, suitable for guiding edge enhancement in thermal infrared images; low-confidence regions may correspond to areas with complex textures or high noise levels, requiring careful handling. Among them, the edge feature map is used to characterize the strength or direction of the probability that each position in the image belongs to an edge; the classic edge detection operator is the core tool for extracting image edges (i.e., gray-level or color change regions), which uses convolution operation to perform gradient calculation or template matching on the gray-level changes of image pixels to highlight regions with significant gray-level differences; the edge features in the confidence calculation based on edge feature strength refer to the core basic features extracted from the data (such as edges in the image), and the strength is the contribution / importance score of these features to the task objective. The calculation logic is: directly compress the strength of the edge features into a confidence score in the 0-1 range through the Sigmoid function. The core is to directly represent the confidence score with the importance of the feature itself.

[0172] Step S702: The features of the high-frequency enhanced image are stitched together with the edge features to generate the original residual image.

[0173] For example, feature maps extracted from the high-frequency enhanced image, such as deep features obtained through one or more convolutional layers, are concatenated with the obtained edge feature map along the channel dimension. The concatenated composite feature map is then input into a small residual prediction network, which outputs a raw residual map. This raw residual map is an unconstrained preliminary prediction that reflects the necessary modification amount perceived by the network based on visible light edge cues. The residual prediction network is a deep learning architecture designed to address the "vanishing / exploding gradients" and "degradation problems" in deep neural network training. It typically consists of several convolutional layers, and its task is to learn a residual map that theoretically contains the pixel-level adjustments needed to match the ideal edge effect in the current high-frequency enhanced image.

[0174] Step S703: Based on physical constraints, the original residual map is modified to obtain a constrained residual map; the physical constraints include zero mean constraint and amplitude limit constraint.

[0175] The zero-mean constraint is achieved by subtracting most of the mean (e.g., 80%) from the original residual map. This aims to eliminate the overall DC offset of the residual map to the greatest extent possible, ensuring that the residual addition operation does not significantly change the average brightness (i.e., the overall temperature level) of the original image, but retains a small portion of the mean to maintain gradient flow for training. The amplitude constraint uses nonlinear functions such as the hyperbolic tangent function to limit the residual values ​​to a reasonable and small range, preventing excessive and unnatural abrupt changes in individual pixels, thereby ensuring enhanced smoothness and naturalness.

[0176] For example, since the original residual map may contain unreasonable adjustments, such as an overall change in the image mean causing temperature drift, or excessive adjustment of certain pixels causing distortion, it is corrected by applying zero-mean constraints and amplitude limiting constraints to maintain the physical authenticity of the thermal infrared image. The constrained residual map is obtained after these two constraint correction steps.

[0177] Step S704: The constraint residual map is weighted by the edge confidence mask to obtain the final residual map.

[0178] The physical meaning of weighted processing is that in regions with high edge confidence (mask value close to 1), residual information is almost completely preserved, allowing for significant edge sharpening; in regions with low edge confidence (mask value close to 0), residual information is strongly suppressed or even completely reduced to zero, thereby preserving the characteristics of the original thermal infrared image in that region to the greatest extent.

[0179] For example, to ensure that the residual thinning operation only applies to reliable edge regions and avoids introducing unnecessary noise or artifacts in smooth regions, each pixel value in the constrained residual map is multiplied by the edge confidence mask value at the corresponding position to obtain the final residual map.

[0180] Step S705: Based on the second attenuable mixing coefficient, the final residual image and the high-frequency enhanced image are mixed to generate a super-resolution thermal infrared image.

[0181] Among them, the second attenuable mixing coefficient As the number of training rounds gradually decreases, in the early stages of training, A larger value means the output tends to retain the stable state before refinement, which helps the network gradually learn reasonable residual predictions; as the training progresses, The value gradually decays to zero according to a predetermined strategy, so that the edge thinning effect is fully effective. This progressive blending strategy acts as a safety valve to avoid the destructive impact of residual prediction errors on image quality during unstable training periods, ensuring robust convergence throughout the training process.

[0182] For example, the final residual image is added pixel-by-pixel to the input high-frequency enhanced image to obtain an initial edge thinning. Subsequently, a second attenuable mixing coefficient is introduced. The preliminary refinement results are then weighted and mixed with the original high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0183] In this embodiment, edge features and confidence masks are extracted from the visible light image. The feature map extracted from the high-frequency enhanced image is then stitched with the edge features to generate a residual map. This residual map is then corrected by applying strict zero-mean and amplitude limiting constraints. Spatial weighting using the confidence mask ensures that the thinning operation is precisely focused on high-confidence edge regions. Finally, a decaying mixing coefficient is introduced to smoothly integrate the thinning effect into the final result, generating a super-resolution thermal infrared image. This method achieves precise, controllable, and secure enhancement of edge details. While significantly improving the edge sharpness of the image, its physical reliability as a thermal infrared image remains unaffected, perfectly achieving a balance between improving visual quality and maintaining physical properties.

[0184] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0185] Based on the same inventive concept, this application also provides a multimodal infrared image super-resolution reconstruction system for implementing the multimodal infrared image super-resolution reconstruction method described above. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal infrared image super-resolution reconstruction system provided below can be found in the limitations of the multimodal infrared image super-resolution reconstruction method described above, and will not be repeated here.

[0186] In one exemplary embodiment, such as Figure 3 As shown, a multimodal infrared image super-resolution reconstruction system 300 is provided, comprising:

[0187] The image registration module 301 is used to acquire a low-resolution thermal infrared image and a corresponding high-resolution visible light image; and to perform spatial registration processing on the high-resolution visible light image based on the low-resolution thermal infrared image to obtain a registered visible light image.

[0188] The feature extraction module 302 is used to perform feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature map and visible light feature map;

[0189] The feature fusion module 303 is used to perform feature-level frequency domain fusion processing on the thermal infrared feature map and the visible light feature map to generate a fused feature map.

[0190] Image enhancement module 304 is used to perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and to perform physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0191] The super-resolution image generation module 305 is used to perform physical consistency edge refinement processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0192] In one embodiment, the feature fusion module 303 is further configured to:

[0193] Fast Fourier transforms were performed on the thermal infrared feature map and the visible light feature map respectively to obtain the thermal infrared frequency domain features and the visible light frequency domain features;

[0194] Amplitude difference gating weights are calculated based on the amplitude spectra of thermal infrared frequency domain characteristics and the amplitude spectra of visible light frequency domain characteristics.

[0195] Phase consistency gating weights are calculated based on the phase spectrum of thermal infrared frequency domain characteristics and the phase spectrum of visible light frequency domain characteristics.

[0196] The fusion weights of visible light features are calculated based on the preset high-frequency mask, amplitude difference gating weights, and phase consistency gating weights; the preset high-frequency mask is used to distinguish between high-frequency and low-frequency regions in the frequency domain.

[0197] Based on the fusion weights of visible light features, the visible light frequency domain features and the thermal infrared frequency domain features are weighted and fused to obtain the fused frequency domain features;

[0198] The fused frequency domain features are subjected to inverse fast Fourier transform to generate a fused feature map.

[0199] In one embodiment, the feature fusion module 303 is further configured to:

[0200] Calculate the absolute difference between the amplitude spectrum of the thermal infrared frequency domain characteristics and the amplitude spectrum of the visible light frequency domain characteristics;

[0201] The absolute difference values ​​are normalized to obtain the normalized magnitude difference.

[0202] Based on the normalized magnitude difference, the first initial gate value is calculated using the first learnable gate function:

[0203]

[0204] in, The first initial gate value, To normalize the difference in magnitude, and These are learnable parameters;

[0205] Based on the preset first calibration parameters, the first initial gate value is subjected to range calibration processing to obtain the functional amplitude difference gate weight; the first calibration parameters include amplitude scaling factor and amplitude offset.

[0206] In one embodiment, the feature fusion module 303 is further configured to:

[0207] Calculate the absolute phase difference between the phase spectrum of the thermal infrared frequency domain characteristics and the phase spectrum of the visible light frequency domain characteristics;

[0208] The phase consistency index is obtained by performing complementation on the absolute phase difference value;

[0209] Based on the phase consistency index, the second initial gate value is calculated using a second learnable gate function:

[0210]

[0211] in, This is the second initial gate value. This is the absolute phase difference value. and These are learnable parameters;

[0212] Based on the preset second calibration parameters, the second initial gate value is subjected to range calibration processing to obtain the phase consistency gate weight; the second calibration parameters include the phase scaling coefficient and the phase offset.

[0213] In one embodiment, the feature fusion module 303 is further configured to:

[0214] A high-frequency mask is generated based on the frequency coordinates; and high-frequency and low-frequency regions are determined based on the high-frequency mask; the high-frequency region is the area of ​​the high-frequency mask that is more than a certain distance from the frequency center than the cutoff frequency; the cutoff frequency is a learnable parameter.

[0215] Based on amplitude difference gating weights, phase consistency gating weights, and fundamental weights, the fusion weights for the high-frequency region are calculated using the following formula:

[0216]

[0217] in, Here, P represents the fusion weight for the high-frequency region, and P represents the phase consistency gating weight. The threshold weights are the amplitude difference gating weights, where clamp is the constraint function, 0.05 is the base weight, (0.05, 0, 999) is the weight value limit range, and 0.95 is the upper limit of the adjustment coefficient.

[0218] Set the fusion weight of the low-frequency region to 0; and merge the fusion weights of the high-frequency region and the low-frequency region to obtain the fusion weight of the visible light features.

[0219] In one embodiment, the image enhancement module 304 is further configured to:

[0220] The registered visible light image is converted into a grayscale image; and the initial super-resolution thermal infrared image and grayscale image are processed by fast Fourier transform to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum and visible light amplitude spectrum.

[0221] The visible light amplitude spectrum is globally normalized to obtain the normalized visible light amplitude spectrum;

[0222] Based on the normalized visible light amplitude spectrum, thermal infrared amplitude spectrum, and a preset high-frequency mask, the enhanced amplitude spectrum in the high-frequency region is calculated using the following formula:

[0223]

[0224] in, The enhanced amplitude spectrum in the high-frequency region, The thermal infrared amplitude spectrum, For learnable enhancement strength coefficients, For the preset high-frequency mask, Normalized visible light amplitude spectrum;

[0225] The thermal infrared amplitude spectrum is determined as the enhanced amplitude spectrum in the low-frequency region; and the enhanced amplitude spectra in the high-frequency region and the enhanced amplitude spectra in the low-frequency region are combined to obtain the enhanced amplitude spectrum.

[0226] An enhanced image is generated by performing an inverse fast Fourier transform on the enhanced amplitude spectrum and the thermal infrared phase spectrum.

[0227] Based on the first attenuable mixing coefficient, the enhanced image is mixed with the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

[0228] In one embodiment, the super-resolution image generation module 305 is further configured to:

[0229] Extracting edge features and edge confidence masks from registered visible light images;

[0230] The features of the high-frequency enhanced image are stitched together with the edge features to generate the original residual image;

[0231] Based on physical constraints, the original residual plot is modified to obtain a constrained residual plot; the physical constraints include zero mean constraint and amplitude limit constraint.

[0232] The constraint residual map is weighted by using an edge confidence mask to obtain the final residual map;

[0233] Based on the second attenuable mixing coefficient, the final residual map and the high-frequency enhanced image are mixed to generate a super-resolution thermal infrared image.

[0234] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the multimodal infrared image super-resolution reconstruction method as described above.

[0235] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0236] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0237] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for super-resolution reconstruction of multimodal infrared images, characterized in that, The method includes: A low-resolution thermal infrared image and a corresponding high-resolution visible light image are acquired; and the high-resolution visible light image is spatially registered using the low-resolution thermal infrared image as a reference to obtain a registered visible light image. The low-resolution thermal infrared image and the registered visible light image are subjected to feature extraction processing to obtain thermal infrared feature maps and visible light feature maps, respectively. The thermal infrared feature map and the visible light feature map are subjected to feature-level frequency domain fusion processing to generate a fused feature map; The fused feature map is convolved to generate an initial super-resolution thermal infrared image; and the initial super-resolution thermal infrared image is subjected to physical consistency high-frequency enhancement processing to obtain a high-frequency enhanced image. The high-frequency enhanced image is subjected to physical consistency edge thinning processing to generate a super-resolution thermal infrared image.

2. The method according to claim 1, characterized in that, The step of performing feature-level frequency domain fusion processing on the thermal infrared feature map and the visible light feature map to generate a fused feature map includes: Fast Fourier transform is performed on the thermal infrared feature map and the visible light feature map respectively to obtain thermal infrared frequency domain features and visible light frequency domain features; Based on the amplitude spectrum of the thermal infrared frequency domain characteristics and the amplitude spectrum of the visible light frequency domain characteristics, the amplitude difference gating weight is calculated; Based on the phase spectrum of the thermal infrared frequency domain characteristics and the phase spectrum of the visible light frequency domain characteristics, the phase consistency gating weight is calculated; The fusion weight of visible light features is calculated based on the preset high-frequency mask, the amplitude difference gating weight, and the phase consistency gating weight; the preset high-frequency mask is used to distinguish between high-frequency and low-frequency regions in the frequency domain. Based on the fusion weights of the visible light features, the visible light frequency domain features and the thermal infrared frequency domain features are weighted and fused to obtain fused frequency domain features; The fused frequency domain features are subjected to inverse fast Fourier transform to generate a fused feature map.

3. The method according to claim 2, characterized in that, The calculation of amplitude difference gating weights based on the amplitude spectrum of the thermal infrared frequency domain characteristics and the amplitude spectrum of the visible light frequency domain characteristics includes: Calculate the absolute difference between the amplitude spectrum of the thermal infrared frequency domain feature and the amplitude spectrum of the visible light frequency domain feature; The absolute difference value is normalized to obtain the normalized magnitude difference. Based on the normalized amplitude difference, the first initial gate value is calculated using the first learnable gate function: in, The first initial gate value, To normalize the difference in magnitude, and These are learnable parameters; Based on the preset first calibration parameters, the first initial gate value is subjected to range calibration processing to obtain the functional amplitude difference gate weight; the first calibration parameters include amplitude scaling factor and amplitude offset.

4. The method according to claim 2, characterized in that, The calculation of phase consistency gating weights based on the phase spectrum of the thermal infrared frequency domain characteristics and the phase spectrum of the visible light frequency domain characteristics includes: Calculate the absolute phase difference between the phase spectrum of the thermal infrared frequency domain feature and the phase spectrum of the visible light frequency domain feature; The absolute phase difference is complemented to obtain the phase consistency index; Based on the phase consistency index, a second initial gate value is calculated using a second learnable gate function: in, This is the second initial gate value. This is the absolute phase difference value. and These are learnable parameters; Based on the preset second calibration parameters, the second initial gate value is subjected to range calibration processing to obtain the phase consistency gate weight; the second calibration parameters include the phase scaling coefficient and the phase offset.

5. The method according to claim 2, characterized in that, The step of calculating the fusion weights of visible light features based on a preset high-frequency mask, the amplitude difference gating weights, and the phase consistency gating weights includes: A high-frequency mask is generated based on the frequency coordinates; and a high-frequency region and a low-frequency region are determined based on the high-frequency mask; the high-frequency region is the area of ​​the high-frequency mask whose distance from the frequency center exceeds the cutoff frequency; the cutoff frequency is a learnable parameter. Based on the amplitude difference gating weight, the phase consistency gating weight, and the basic weight, the fusion weight of the high-frequency region is calculated using the following formula: in, Here, P represents the fusion weight for the high-frequency region, and P represents the phase consistency gating weight. The threshold weights are the amplitude difference gating weights, where clamp is the constraint function, 0.05 is the base weight, (0.05, 0, 999) is the weight value limit range, and 0.95 is the upper limit of the adjustment coefficient. The fusion weight of the low-frequency region is set to 0; and the fusion weight of the high-frequency region and the fusion weight of the low-frequency region are merged to obtain the fusion weight of the visible light features.

6. The method according to claim 1, characterized in that, The step of performing physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image includes: The registered visible light image is converted into a grayscale image; and the initial super-resolution thermal infrared image and the grayscale image are processed by fast Fourier transform to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum and visible light amplitude spectrum. The visible light amplitude spectrum is globally normalized to obtain the normalized visible light amplitude spectrum; Based on the normalized visible light amplitude spectrum, the thermal infrared amplitude spectrum, and the preset high-frequency mask, the enhanced amplitude spectrum in the high-frequency region is calculated using the following formula: in, The enhanced amplitude spectrum in the high-frequency region, The thermal infrared amplitude spectrum, For learnable enhancement strength coefficients, For the preset high-frequency mask, Normalized visible light amplitude spectrum; The thermal infrared amplitude spectrum is determined as the enhanced amplitude spectrum in the low-frequency region; and the enhanced amplitude spectrum in the high-frequency region and the enhanced amplitude spectrum in the low-frequency region are combined to obtain the enhanced amplitude spectrum. Perform inverse fast Fourier transform on the enhanced amplitude spectrum and the thermal infrared phase spectrum to generate an enhanced image; Based on the first attenuable mixing coefficient, the enhanced image is mixed with the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image.

7. The method according to claim 1, characterized in that, The step of performing physical consistency edge thinning processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image includes: Edge features and edge confidence masks are extracted from the registered visible light image; The features of the high-frequency enhanced image are concatenated with the edge features to generate the original residual image; Based on physical constraints, the original residual map is modified to obtain a constrained residual map; the physical constraints include zero mean constraint and amplitude limiting constraint. The constraint residual map is weighted using the edge confidence mask to obtain the final residual map; Based on the second attenuable mixing coefficient, the final residual image and the high-frequency enhanced image are mixed to generate a super-resolution thermal infrared image.

8. A multimodal infrared image super-resolution reconstruction system, characterized in that, The system includes: An image registration module is used to acquire a low-resolution thermal infrared image and a corresponding high-resolution visible light image; and to perform spatial registration processing on the high-resolution visible light image based on the low-resolution thermal infrared image to obtain a registered visible light image. The feature extraction module is used to perform feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature map and visible light feature map; The feature fusion module is used to perform feature-level frequency domain fusion processing on the thermal infrared feature map and the visible light feature map to generate a fused feature map. The image enhancement module is used to perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and to perform physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image. The super-resolution image generation module is used to perform physical consistency edge refinement processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images