A wavelet domain infrared image super-resolution reconstruction method, system, device and medium

Abstract

This application relates to a method, system, device, and medium for super-resolution reconstruction of multimodal infrared images based on wavelet domain trust gating. The method includes: acquiring a low-resolution thermal infrared image and a corresponding high-resolution visible light image and performing spatial registration; extracting thermal infrared feature maps and visible light feature maps respectively; performing wavelet domain dual-track trust fusion on the two feature maps: calculating trust weights per spatial location based on local phase consistency and local amplitude differences across multiple scales and directions using dual-tree complex wavelet transform, mapping these weights to a sub-band grid of discrete wavelet transform to complete weighted fusion, and generating a fused feature map through inverse transform; performing convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image, and then generating the final super-resolution thermal infrared image. This method implements localized trust adjudication per scale, per direction, and per spatial location, suppressing cross-modal texture leakage while introducing visible light details, and exhibiting robustness to registration residuals.

Description

Technical Field

[0003] This invention belongs to the field of image super-resolution reconstruction technology, and in particular relates to a method, system, device and medium for multimodal infrared image super-resolution reconstruction based on wavelet domain trust gating. Background Technology

[0005] With the widespread application of infrared imaging technology in security monitoring, industrial inspection, and medical diagnosis, increasingly higher demands are being placed on the spatial resolution of infrared images. However, limited by the materials, processes, and physical diffraction limits of infrared detectors, acquiring high spatial resolution infrared images is typically costly and technically complex. Therefore, using algorithms to perform super-resolution reconstruction of low-resolution infrared images has become an economical and effective technical approach. In recent years, with the development of multimodal sensing technology, the idea of ​​fusing high-resolution visible light images to help improve the resolution of infrared images has received widespread attention.

[0006] Traditional techniques for super-resolution reconstruction of multimodal images mainly fall into two categories. The first category involves direct spatial fusion, such as concatenating registered visible light images with upsampled infrared images and then inputting the concatenated images into a deep convolutional neural network, or employing attention-based fusion strategies to adaptively select visible light details in the spatial domain. These methods rely on the network learning intermodal relationships within spatial features. However, since infrared images reflect the thermal radiation characteristics of objects while visible light images reflect their reflectivity, and these two images differ fundamentally in grayscale distribution and edge characteristics, direct spatial fusion is prone to introducing artifacts due to physical property mismatch, or causing crucial thermal information to be obscured by visible light intensity textures.

[0007] The second type of method transfers the fusion process to the transform domain, for example, by performing a global frequency domain transformation (such as a fast Fourier transform) on the feature map as a whole, and defining fusion weights on the global frequency components, using the frequency domain consistency between modes as the criterion for introducing visible light information. While such methods alleviate the modal conflict problem in spatial domain fusion to some extent, they still have the following inherent drawbacks: First, each frequency component of the global transformation corresponds to a basis function covering the entire image, and the fusion weight is a unified decision for the entire image regarding that frequency component, lacking spatial selectivity. On the same frequency component, a real cross-modal shared structure in one part of the image and a visible light-specific texture (such as sign text or advertising patterns) are superimposed in the same decision and diluted with each other, making it impossible to fully introduce reliable local details while suppressing local texture leakage. Second, the phase comparison in the global transform domain is sensitive to the local registration residuals between modes. In actual systems, there are generally local misalignments and non-rigid offsets between thermal infrared sensors and visible light sensors that are difficult to completely eliminate. Such local misalignments perturb the phase of all frequency components in a distributed manner in the global transform domain, reducing the reliability of the phase consistency criterion. Third, the global transformation usually needs to complete all fusion operations in the complex domain, and the judgment and execution are coupled on the same computationally expensive transformation path, which is not conducive to the deployment of resource-constrained platforms.

[0008] Therefore, there is an urgent need for a multimodal infrared image super-resolution reconstruction method that can perform scale-wise, direction-wise, and spatial location-wise localized trust decisions on visible light guidance information, is robust to intermodal registration residuals, and has computationally decoupled decision-making and execution. Summary of the Invention

[0010] Therefore, it is necessary to provide a super-resolution reconstruction method that can effectively introduce high-frequency details of visible light images at the local scale, suppress cross-modal texture leakage, and strictly maintain the physical properties of infrared images to address the above-mentioned technical problems.

[0011] In a first aspect, this application provides a multimodal infrared image super-resolution reconstruction method based on wavelet domain trust gating, including:

[0012] Acquire a low-resolution thermal infrared image and its corresponding high-resolution visible light image; and use the low-resolution thermal infrared image as a reference to perform spatial registration processing on the high-resolution visible light image to obtain a registered visible light image.

[0013] Feature extraction processing was performed on the low-resolution thermal infrared image and the registered visible light image to obtain thermal infrared feature map and visible light feature map, respectively.

[0014] The thermal infrared feature map and the visible light feature map are subjected to dual-tree complex wavelet transform to obtain thermal infrared complex wavelet coefficients and visible light complex wavelet coefficients respectively. Both thermal infrared complex wavelet coefficients and visible light complex wavelet coefficients are organized according to decomposition level and direction sub-band, and each coefficient contains local amplitude and local phase.

[0015] Based on the local phase difference and local amplitude difference of thermal infrared complex wavelet coefficients and visible light complex wavelet coefficients at corresponding levels, directions, and spatial locations, the trust weights are calculated for each level, direction, and spatial location.

[0016] Discrete wavelet transforms were performed on the thermal infrared feature map and the visible light feature map respectively to obtain their respective low-frequency sub-bands and multi-directional high-frequency sub-bands;

[0017] Trust weights are mapped to the high-frequency subband grid of discrete wavelet transform, and based on the mapped trust weights, the high-frequency subbands of the visible light feature map and the high-frequency subbands of the thermal infrared feature map are weighted and fused to obtain the fused high-frequency subband; the low-frequency subbands of the thermal infrared feature map and the low-frequency subbands of the visible light feature map are fused to obtain the fused low-frequency subband.

[0018] Inverse discrete wavelet transform is performed on the fused low-frequency subband and the fused high-frequency subband to generate a fused feature map;

[0019] The fused feature map is convolved to generate an initial super-resolution thermal infrared image; and a super-resolution thermal infrared image is generated based on the initial super-resolution thermal infrared image.

[0020] Furthermore, based on the local phase and amplitude differences of the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients at corresponding levels, directions, and spatial locations, trust weights are calculated for each level, direction, and spatial location, including:

[0021] The absolute difference between the local amplitude of the thermal infrared complex wavelet coefficients and the local amplitude of the visible light complex wavelet coefficients is calculated, and the absolute difference is normalized to obtain the normalized amplitude difference. Based on the normalized amplitude difference, the amplitude difference gating weight is calculated through the first learnable gating function.

[0022] The absolute phase difference between the local phase of the thermal infrared complex wavelet coefficients and the local phase of the visible light complex wavelet coefficients is calculated, and the absolute phase difference is subjected to angle wrapping and normalization to obtain the local phase consistency index. Based on the local phase consistency index, the phase consistency gating weight is calculated through a second learnable gating function.

[0023] Trust weights are calculated based on amplitude difference gating weights, phase consistency gating weights, and base weights; the base weights are greater than zero, ensuring that visible light contributions at any level, in any direction, and at any spatial location are not completely shut off.

[0024] Furthermore, mapping the trust weights to the high-frequency subband grid of the discrete wavelet transform includes: pairing and merging the six directional subbands of the dual-tree complex wavelet transform into three sets of directional trust weights that correspond one-to-one with the three high-frequency directional subbands of the discrete wavelet transform according to the symmetrical pairing relationship of the directional angles; when the spatial grid size of the merged trust weights is inconsistent with the spatial grid size of the high-frequency subband of the corresponding level of the discrete wavelet transform, the merged trust weights are aligned to the spatial grid of the high-frequency subband through spatial interpolation.

[0025] Furthermore, the method also includes hierarchical emphasis processing: setting learnable hierarchical emphasis parameters for each feature channel and converting the hierarchical emphasis parameters into hierarchical emphasis coefficients through bounded mapping; based on the hierarchical emphasis coefficients, applying complementary hierarchical weights to the trust weights of different decomposition levels, so that different wavelet domain trust fusion units are differentiated into scale divisions emphasizing different decomposition levels during training.

[0026] Furthermore, the low-frequency sub-bands of the thermal infrared feature map and the low-frequency sub-bands of the visible light feature map are fused, including: calculating the normalized difference between the two low-frequency sub-bands, calculating the low-frequency thermal infrared weights according to the thermal infrared dominance principle, so that the fused low-frequency sub-band is dominated by the thermal infrared low-frequency components, so as to maintain the overall thermal radiation distribution characteristics of the thermal infrared image.

[0027] Furthermore, the method also includes cross-directional coherence correction processing: performing neighborhood aggregation on the phase consistency degree of adjacent directional sub-bands within the same decomposition level to obtain a cross-directional coherence index; and correcting the trust weight based on the cross-directional coherence index.

[0028] Furthermore, based on the initial super-resolution thermal infrared image, a super-resolution thermal infrared image is generated, including: performing physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image; and performing physical consistency edge thinning processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0029] Secondly, this application also provides a multimodal infrared image super-resolution reconstruction system based on wavelet domain trust gating, comprising:

[0030] The image registration module is used to acquire a low-resolution thermal infrared image and the corresponding high-resolution visible light image; and to perform spatial registration processing on the high-resolution visible light image based on the low-resolution thermal infrared image to obtain a registered visible light image.

[0031] The feature extraction module is used to perform feature extraction processing on low-resolution thermal infrared images and registered visible light images respectively, to obtain thermal infrared feature maps and visible light feature maps;

[0032] The wavelet domain trust fusion module is used to perform dual-tree complex wavelet transforms on the thermal infrared feature map and the visible light feature map respectively, and calculate trust weights based on the local phase differences and local amplitude differences at the corresponding level, corresponding direction, and corresponding spatial location; it also performs discrete wavelet transforms on the thermal infrared feature map and the visible light feature map respectively, maps the trust weights to the high-frequency sub-band grid of the discrete wavelet transform and completes weighted fusion, and generates a fused feature map through inverse discrete wavelet transform;

[0033] The super-resolution image generation module is used to perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and to generate a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement any of the wavelet domain trust-gated multimodal infrared image super-resolution reconstruction methods described in the embodiments of this application.

[0035] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement a wavelet domain trust-gated multimodal infrared image super-resolution reconstruction method as described in any of the embodiments of this application.

[0036] The aforementioned method, system, device, and medium for multimodal infrared image super-resolution reconstruction based on wavelet domain trust gating decouples trust judgment and fusion execution onto two wavelet transforms: trust judgment is undertaken by a dual-tree complex wavelet transform with approximately translation invariance, which uses local phase consistency and local amplitude difference to jointly determine the credibility of visible light information on multi-scale and multi-directional local support; fusion execution is undertaken by a computationally inexpensive and perfectly reconstructable discrete wavelet transform, which applies hierarchical, directional, and spatial location-based trust weights to the weighted fusion of real subbands. The beneficial effects achieved include: First, the trust decision is spatially selective; at the same scale and in the same direction, credible shared structures and untrusted modal-specific textures obtain independent decisions at different spatial locations, which can fully introduce local credible details while suppressing cross-modal texture leakage. Second, the dual-tree complex wavelet phase has an approximately linear response to local translation, and the trust criterion is robust to the registration residuals between modalities. Third, the decision track and the execution track are computationally decoupled; the execution track only involves real-number discrete wavelet transform and element-wise weighting, resulting in low computational overhead and facilitating deployment on resource-constrained platforms. Fourth, the hierarchical emphasis mechanism enables multiple stacked fusion units to spontaneously differentiate into different scales of specialization, and experimental observations show that wavelet domain trust gating exhibits a block-to-block specialization pattern and training dynamics that are qualitatively different from global frequency domain gating, proving that the localized trust mechanism of this invention is not a simple transformation or replacement of the global frequency domain scheme. Fifth, the low-frequency sub-bands are fused according to the thermal infrared dominance principle, strictly maintaining the overall thermal radiation distribution characteristics of the thermal infrared image, meeting the physical authenticity requirements of quantitative thermal analysis applications. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a multimodal infrared image super-resolution reconstruction method based on wavelet domain trust gating in one embodiment.

[0040] Figure 2 This is a schematic diagram of the steps in one embodiment to perform wavelet domain dual-track trust fusion processing on thermal infrared feature maps and visible light feature maps to generate a fused feature map.

[0041] Figure 3 This is a schematic diagram illustrating the pairing and merging relationship between the six-directional subbands of dual-tree complex wavelet transform and the three-directional high-frequency subbands of discrete wavelet transform in one embodiment.

[0042] Figure 4 This is a schematic diagram of the structure of a multimodal infrared image super-resolution reconstruction system based on wavelet domain trust gating in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In one embodiment, a multimodal infrared image super-resolution reconstruction method based on wavelet domain trust gating is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and can be implemented through the interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0046] Step S101: Acquire a low-resolution thermal infrared image and the corresponding high-resolution visible light image; and use the low-resolution thermal infrared image as a reference to perform spatial registration processing on the high-resolution visible light image to obtain a registered visible light image.

[0047] Spatial registration refers to unifying spatial data such as images from different sources, at different times, with different resolutions, or in different coordinate systems into the same standard spatial reference frame, eliminating positional deviations between data, and ensuring that they correspond accurately in spatial location.

[0048] For example, a low-resolution thermal infrared image and a corresponding high-resolution visible light image of the same scene are acquired using an image acquisition device. A feature-point-based registration method can be used, with the low-resolution thermal infrared image as the spatial reference, to calculate a geometric transformation model (such as an affine transformation) from the visible light image to the thermal infrared image, and then use this model to resample the visible light image to obtain the registered visible light image. It should be noted that in actual systems, registration processing usually cannot completely eliminate local misalignment and non-rigid offset. The trust criterion in subsequent steps of this application is robust to such registration residuals, so the registration accuracy requirement in this step can be appropriately relaxed.

[0049] Step S102: Perform feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature map and visible light feature map.

[0050] Feature extraction refers to the step of extracting key information that is meaningful to the target task from raw data such as images. The purpose is to transform the pixel information of the raw image into a high-dimensional feature representation that can better represent the content of the image.

[0051] For example, for low-resolution thermal infrared images, they can first be upsampled (e.g., bicubic interpolation) to the target resolution, and then thermal infrared feature maps can be extracted using a lightweight convolutional encoder; for registered visible light images, visible light feature maps can be extracted using a feature extraction network containing residual blocks or densely connected blocks. Both feature extraction networks must ensure that the output thermal infrared and visible light feature maps are consistent in spatial size and number of channels.

[0052] Step S103: Perform wavelet domain dual-track trust fusion processing on the thermal infrared feature map and the visible light feature map to generate a fused feature map.

[0053] Among them, wavelet domain dual-track trust fusion refers to a fusion mechanism that decouples trust judgment and fusion execution onto two wavelet transforms: the trust track uses dual-tree complex wavelet transform to calculate the credibility of visible light information on multi-scale, multi-directional local supports; the execution track uses discrete wavelet transform to apply trust weights to the weighted fusion of real subbands. Dual-tree complex wavelet transform (DT-CWT) is a complex wavelet transform that is approximately translation-invariant and has six directional selective subbands. The magnitude of each coefficient represents the energy intensity of the local directional structure, and the phase encodes the precise positional relationship of the structure within the wavelet support. Discrete wavelet transform (DWT) is a real-number, decimation-based, perfectly reconstructable multi-resolution decomposition. Each decomposition level generates a low-frequency subband and three directional (horizontal, vertical, and diagonal) high-frequency subbands.

[0054] For example, wavelet domain dual-track trust fusion processing is performed on the thermal infrared feature map and the visible light feature map. The generated fused feature map introduces detailed visible light information at locally trustworthy locations, while retaining the original content of the thermal infrared image at untrustworthy locations. Detailed steps are described in the embodiments below. Figure 2 .

[0055] Step S104: Perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image.

[0056] For example, the fused feature map is adjusted for channel number and integrated through a convolutional layer (typically a 1x1 convolution), and restored to the target resolution through a progressive upsampling path to generate an initial super-resolution thermal infrared image.

[0057] Step S105: Generate a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image.

[0058] For example, the initial super-resolution thermal infrared image can be directly output as a super-resolution thermal infrared image; alternatively, the initial super-resolution thermal infrared image can be further processed by physical consistency high-frequency enhancement and physical consistency edge refinement before output, as detailed in the embodiments described below.

[0059] In this embodiment, multimodal images are acquired and registered. Feature extraction is performed on the registered images, and the feature maps are fused using wavelet domain dual-track trust fusion to generate a fused feature map. This fused feature map is then processed by convolution and reconstruction to obtain a super-resolution thermal infrared image. The entire process utilizes a localized trust adjudication mechanism to adaptively introduce visible light details on a scale-by-scale, direction-by-direction, and spatial-position basis. This effectively overcomes the modal conflict problem of spatial domain fusion and the lack of spatial selectivity and sensitivity to registration residuals in global frequency domain fusion.

[0060] In one embodiment, such as Figure 2 As shown, wavelet domain dual-track trust fusion processing is performed on the thermal infrared feature map and the visible light feature map to generate a fused feature map, including:

[0061] Step S201: Perform dual-tree complex wavelet transform on the thermal infrared feature map and the visible light feature map respectively to obtain the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients.

[0062] Among them, the dual-tree complex wavelet transform constructs complex coefficients through two parallel real wavelet filter trees, overcoming the defects of ordinary discrete wavelet transform such as translation sensitivity and poor direction selectivity; it generates six directional sub-bands at each decomposition level, with directional angles of approximately 15 degrees, 45 degrees, 75 degrees, 105 degrees, 135 degrees and 165 degrees, and each spatial position within each sub-band corresponds to a complex coefficient.

[0063] For example, J-level dual-tree complex wavelet transforms (e.g., J=2) are performed on the thermal infrared feature map and the visible light feature map respectively by channel. The first-level filter can be a near-symmetric bioorthogonal filter bank (e.g., near_sym_a), and the second and subsequent levels can be a quarter-sample shift orthogonal filter bank (e.g., qshift_a). Before the transform, the feature map can be filled with reflections to make its spatial size an integer multiple of 2 to the power of J, to ensure the regularity of the decomposition. The thermal infrared and visible light complex wavelet coefficients obtained by the transform are organized according to (hierarchy, direction, spatial position). The local amplitude of each coefficient is obtained by complex modulus taking, and the local phase is obtained by complex argument taking. To avoid gradient singularities at zero complex number in the amplitude and argument operations, a very small imaginary part bias (e.g., 1e-8) can be applied to the complex coefficients before modulus taking and argument taking.

[0064] Step S202: Calculate the normalized amplitude difference between the local amplitude of the thermal infrared complex wavelet coefficients and the local amplitude of the visible light complex wavelet coefficients, and calculate the amplitude difference gating weight through the first learnable gating function.

[0065] Among them, the local amplitude difference reflects the degree of inconsistency of the directional structural energy of the two modes at this level, in this direction, and at this spatial location: the location with a large difference suggests that visible light has structural components that are lacking in thermal infrared, and there is potential value in introducing visible light information.

[0066] For example, for each level, each direction, and each spatial location, the absolute difference value of the local amplitudes of the two modes is calculated, and then divided by the sum of the maximum amplitude difference within that sub-band and the stability constant to obtain the normalized amplitude difference ΔM_norm, which is then limited to a preset range (e.g., [0,10]). Subsequently, the amplitude difference gating weight is calculated using a first learnable gating function.

[0067] M_H = ( z_m / (1+|z_m|) + 1 ) / 2 , where z_m = β_m·(ΔM_norm − ρ_m)

[0068] Where M_H is the amplitude difference gating weight, β_m is a learnable kurtosis parameter used to control the sensitivity of the gating value to differences, and ρ_m is a learnable threshold parameter used to determine the center point of the gating response; z / (1+|z|) is a soft sign function with S-shaped curve characteristics, which can smoothly compress the input to the (-1,1) interval, and after linear transformation, the output falls in the (0,1) interval. β_m and ρ_m are optimized through backpropagation algorithm during training. To ensure training stability, β_m can be constrained to a preset interval (e.g., constrained to [4,16] through Sigmoid mapping), and ρ_m can be constrained to a preset interval (e.g., [0.01,0.15]).

[0069] Step S203: Calculate the local phase consistency index between the local phase of the thermal infrared complex wavelet coefficients and the local phase of the visible light complex wavelet coefficients, and calculate the phase consistency gating weight through the second learnable gating function.

[0070] The local phase encoding of the dual-tree complex wavelet coefficients involves the sub-pixel positions and arrangement relationships within the wavelet support. A real physical structure (such as a pedestrian outline or vehicle boundary) leaves phase-aligned imprints at corresponding positions in both modes, while visible light-specific textures (such as sign text) lack phase counterparts in the thermal infrared coefficients. Local phase consistency thus constitutes a localized criterion for the reliability of visible light structural information. Unlike the phase of a global frequency domain transform, the dual-tree complex wavelet phase exhibits an approximately linear response to local translations; therefore, this criterion is robust to local registration residuals between modes.

[0071] For example, for each level, each direction, and each spatial location, the local phase difference between the two modes is calculated, and the sine and cosine components are processed using the arctangent function to achieve angle wrapping (i.e., the difference is constrained to between -π and π), and then normalized by dividing by π to obtain the normalized phase difference Δφ. Using 1−|Δφ| as the local phase consistency index, the phase consistency gating weight is calculated using a second learnable gating function:

[0072] P = ( z_p / (1+|z_p|) + 1 ) / 2 , where z_p = β_p·( (1 − |Δφ|) − ρ_p )

[0073] Where P is the phase consistency gating weight, β_p is the learnable kurtosis parameter, and ρ_p is the learnable threshold parameter, and the constraint method is the same as in step S202.

[0074] Step S204: Based on the amplitude difference gating weight, phase consistency gating weight, and basic weight, calculate the trust weights for each level, direction, and spatial location.

[0075] For example, the trust weight is calculated using the following formula:

[0076] w = clamp( w_0 + k·P·M_H , w_0 , w_max )

[0077] Where w is the trust weight, w_0 is the base weight greater than zero (e.g., 0.01), k is the adjustment coefficient (e.g., 0.98), w_max is the upper limit of the weight (e.g., 0.99), and clamp is the constraint function. The physical meaning of this formula is: visible light information only receives a high trust weight when there is both an introduceable structural energy difference at a certain location (higher M_H) and the two modes reach a consensus on the local geometry of the structure (higher P); the base weight w_0 ensures that the visible light contribution at any location is not completely shut off, so as to maintain gradient flow and training stability.

[0078] Step S205: Perform discrete wavelet transform on the thermal infrared feature map and the visible light feature map respectively to obtain their respective low-frequency sub-bands and multi-directional high-frequency sub-bands.

[0079] For example, J-level discrete wavelet transforms are performed on the thermal infrared feature map and the visible light feature map respectively by channel. The wavelet basis can be a compactly supported orthogonal wavelet (such as db3), and the boundary extension method can be zero extension. Each level yields three high-frequency subbands in three directions (horizontal, vertical, and diagonal), and the deepest level also yields a low-frequency subband.

[0080] Step S206: Map the trust weights to the high-frequency subband grid of the discrete wavelet transform, and perform weighted fusion of the high-frequency subbands of the two modes based on the mapped trust weights; at the same time, fuse the low-frequency subbands according to the thermal infrared dominance principle.

[0081] Among them, there is an approximate correspondence between the six directional subbands of the dual-tree complex wavelet transform and the three high-frequency directional subbands of the discrete wavelet transform in terms of directional angles: the subband pairs with directional angles symmetric about the horizontal axis (approximately 15 degrees and 165 degrees) jointly represent the near-horizontal structure and correspond to the horizontal high-frequency subbands of the discrete wavelet transform; the subband pairs with directional angles symmetric about the vertical axis (approximately 75 degrees and 105 degrees) jointly represent the near-vertical structure and correspond to the vertical high-frequency subbands; and the diagonal directional subband pairs (approximately 45 degrees and 135 degrees) correspond to the diagonal high-frequency subbands.

[0082] For example, such as Figure 3 As shown, the six sets of directional trust weights are averaged pairwise according to the aforementioned symmetrical pairing relationship and merged into three sets of directional trust weights. Since the spatial grid sizes of the dual-tree complex wavelet transform and the discrete wavelet transform at the same decomposition level may differ due to different boundary extension methods, when the sizes are inconsistent, bilinear interpolation is used to align the merged trust weights to the spatial grid of the corresponding high-frequency subband of the discrete wavelet transform level, and a constraint function is applied again to ensure that the weights are within the effective range. Subsequently, for each level, each direction, and each spatial location, high-frequency subband weighted fusion is performed according to the following formula:

[0083] Fusion high-frequency subband = w·visible high-frequency subband + (1−w)·thermal infrared high-frequency subband

[0084] Where w represents the mapped trust weights. Positions with high trust weights emphasize the contribution of the visible light subband to introduce detail, while positions with low trust weights preserve the original content of the thermal infrared subband to maintain physical properties. Since the trust weights are a smooth field in space, the approximation error introduced by directional pairing and merging with spatial interpolation is negligible.

[0085] For the low-frequency subband, calculate and normalize the absolute difference between the two modes of the low-frequency subband to obtain the normalized low-frequency difference ΔL_norm. Calculate the low-frequency thermal infrared weight using the following formula:

[0086] w_th = clamp( w_0 + k·(1 − ΔL_norm) , w_0 , w_th_max )

[0087] Where w_th_max is an upper limit close to 1 (e.g., 0.999). The fused low-frequency subband = (1−w_th)·visible light low-frequency subband + w_th·thermal infrared low-frequency subband. The low-frequency subband dominates the overall brightness distribution and macroscopic thermal contrast, directly corresponding to the most critical temperature characterization information in the thermal infrared image. Fusion according to the thermal infrared-dominated principle can strictly maintain the overall thermal radiation distribution characteristics of the thermal infrared image.

[0088] Step S207: Perform inverse discrete wavelet transform on the fused low-frequency subband and the fused high-frequency subband to generate a fused feature map.

[0089] For example, inverse discrete wavelet transform is performed on the fused low-frequency subband and the fused high-frequency subband at each level to reconstruct the spatial fusion feature map, and the boundary parts filled for regularity before the transform are removed. After reconstruction, numerical cleaning (replacing non-numerical and infinite values ​​with zeros) and channel-wise hybrid convolution (1x1 convolution initialized with an identity matrix for cross-channel information integration) can be applied, and the output is limited to a preset numerical range to ensure the numerical stability of subsequent networks.

[0090] In this embodiment, the credibility of visible light information is jointly determined by local phase consistency and local amplitude difference on multi-scale and multi-directional local supports through dual-tree complex wavelet transform. The trust weights are then mapped to the real sub-bands of the discrete wavelet transform to complete weighted fusion. This approach enables independent judgments on credible structures and untrusted textures at different spatial locations in the image at the same scale and in the same direction, effectively solving the problem of dilution of credible and untrusted information within the same frequency component in global frequency domain fusion. Furthermore, the trust judgment is handled by the approximately translation-invariant complex wavelet phase, which is robust to registration residuals, and the fusion execution is handled by the computationally inexpensive real discrete wavelet transform, facilitating deployment on resource-constrained platforms.

[0091] In one embodiment, the method further includes hierarchical emphasis processing.

[0092] Hierarchical emphasis refers to applying complementary hierarchical weights to the trust weights of different decomposition levels in order to achieve scale division of labor among multiple stacked wavelet domain trust fusion units.

[0093] For example, a learnable hierarchical emphasis parameter is set for each feature channel, and it is mapped to a preset bounded interval (e.g., [0.15, 0.60]) using the Sigmoid function, and then linearly transformed into a hierarchical emphasis coefficient λ with values ​​in [0, 1]. For a two-level decomposition (J=2), the trust weights of fine-level layers are multiplied by λ, and the trust weights of coarse-level layers are multiplied by (1−λ); for more layers, the hierarchical weights can be linearly interpolated between λ and (1−λ) according to the hierarchical index. A lower bound greater than zero (e.g., 0.10) is applied to each hierarchical weight to ensure that no decomposition level is completely shut down. When multiple wavelet domain trust fusion units are stacked in the network, the hierarchical emphasis parameters of different units can be differentially initialized (e.g., the first unit is biased towards fine-level layers, and the second unit is biased towards coarse-level layers), and a differentiation regularization term can be added to enable each unit to spontaneously maintain complementary scale division of labor during training.

[0094] In this embodiment, a learnable hierarchical emphasis mechanism enables the division of labor among multiple fusion units at the decomposition level, allowing different units to handle the introduction of fine-scale details and coarse-scale structures respectively. Experimental observations show that under this mechanism, the gating steepness parameter of fine-level units increases with training while that of coarse-level units remains flat. Furthermore, the visible light admission energy of fine-level units is significantly higher than that of corresponding units in the global frequency domain gating scheme. This inter-block division of labor mode exhibits a qualitative difference from the global frequency domain gating scheme, indicating that the localized trust mechanism of this invention has independent technical effects distinct from the global frequency domain scheme.

[0095] In one embodiment, the method further includes cross-directional coherence correction processing.

[0096] Cross-directional coherence refers to the phase consistency between modes of a real directional structure in its adjacent directional subbands of excitation. If the phase consistency in a certain direction is seriously inconsistent with that in the adjacent directions and is isolated and abnormal, the response is more likely to be caused by aliasing or noise rather than a real structure.

[0097] For example, the phase consistency index of each directional sub-band within the same decomposition level is aggregated with the phase consistency indices of its adjacent directional sub-bands (e.g., by taking the neighborhood mean or neighborhood minimum) to obtain a cross-directional coherence index. This index is then used as a correction factor for the trust weights, resulting in relatively higher trust weights for spatial locations where phase consistency in adjacent directions mutually corroborates each other, and relatively lower trust weights for spatial locations where phase consistency is isolated and anomalous. This correction process does not introduce additional learnable parameters, thus maintaining the interpretability of the trust criterion.

[0098] In this embodiment, cross-directional coherence correction adds a second geometric self-consistency check to the trust decision, which can further suppress misjudgments caused by aliasing or noise and improve the reliability of the introduction of visible light information.

[0099] In one embodiment, a physically consistent high-frequency enhancement process is performed on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image, including:

[0100] The registered visible light image is converted into a grayscale image; then, the initial super-resolution thermal infrared image and the grayscale image are processed by Fast Fourier Transform to obtain the thermal infrared amplitude spectrum, thermal infrared phase spectrum, and visible light amplitude spectrum; the visible light amplitude spectrum is globally normalized to obtain the normalized visible light amplitude spectrum; based on the normalized visible light amplitude spectrum, the thermal infrared amplitude spectrum, and a preset high-frequency mask, the enhanced amplitude spectrum in the high-frequency region is calculated using the following formula:

[0101] M_f = M_t × ( 1 + α·mask_HF·(M_r_norm − 1) )

[0102] Where M_f is the enhanced amplitude spectrum in the high-frequency region, M_t is the thermal infrared amplitude spectrum, α is a learnable enhancement intensity coefficient, mask_HF is a preset high-frequency mask, and M_r_norm is the normalized visible light amplitude spectrum; the thermal infrared amplitude spectrum is determined as the enhanced amplitude spectrum in the low-frequency region, and the enhanced amplitude spectra of the high-frequency region and the low-frequency region are merged to obtain the enhanced amplitude spectrum; the enhanced amplitude spectrum and the thermal infrared phase spectrum are subjected to inverse fast Fourier transform to generate an enhanced image; based on the first attenuable mixing coefficient, the enhanced image is mixed with the initial super-resolution thermal infrared image to obtain the high-frequency enhanced image.

[0103] Among them, strictly maintaining the thermal infrared phase spectrum and the low-frequency amplitude spectrum is a key measure to ensure physical consistency: phase information determines the position of frequency components in the spatial domain, and low-frequency amplitude dominates the overall brightness distribution and macroscopic thermal contrast; the first attenuable mixing coefficient gradually decreases to zero with the number of training rounds as a progressive regularization method to avoid the impact of the initial poor output of the enhancement module on network training.

[0104] In one embodiment, physical consistency edge refinement is performed on the high-frequency enhanced image to generate a super-resolution thermal infrared image, including:

[0105] Edge features and edge confidence masks are extracted from the registered visible light image; the features of the high-frequency enhanced image are stitched together with the edge features to generate the original residual map; the original residual map is corrected based on physical constraints to obtain a constrained residual map, which includes zero-mean constraints and amplitude limiting constraints; the constrained residual map is weighted using the edge confidence mask to obtain the final residual map; based on the second attenuable mixing coefficient, the final residual map and the high-frequency enhanced image are mixed to generate a super-resolution thermal infrared image.

[0106] Among them, the zero-mean constraint is achieved by subtracting most of the mean from the original residual map, which is used to eliminate the overall DC offset of the residual and ensure that the addition of residual does not change the average brightness of the image (i.e., the overall temperature level); the amplitude limiting constraint uses nonlinear functions such as hyperbolic tangent to limit the residual value within a reasonable range to prevent excessive abrupt changes at the pixel level; the edge confidence mask makes the thinning operation only apply to the reliable edge region; the second attenuable mixing coefficient has the same function as the first attenuable mixing coefficient.

[0107] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0108] Based on the same inventive concept, this application also provides a system for implementing the aforementioned wavelet domain trust-gated multimodal infrared image super-resolution reconstruction method. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations in one or more system embodiments provided below can be found in the limitations of the method described above, and will not be repeated here.

[0109] In one exemplary embodiment, such as Figure 4 As shown, a multimodal infrared image super-resolution reconstruction system 400 based on wavelet domain trust gating is provided, comprising:

[0110] The image registration module 401 is used to acquire a low-resolution thermal infrared image and the corresponding high-resolution visible light image; and to perform spatial registration processing on the high-resolution visible light image based on the low-resolution thermal infrared image to obtain a registered visible light image.

[0111] The feature extraction module 402 is used to perform feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature map and visible light feature map;

[0112] The wavelet domain trust fusion module 403 is used to perform dual-tree complex wavelet transform on the thermal infrared feature map and the visible light feature map respectively, and calculate trust weights based on the local phase difference and local amplitude difference at the corresponding level, corresponding direction, and corresponding spatial position; to perform discrete wavelet transform on the thermal infrared feature map and the visible light feature map respectively, to map the trust weights to the high-frequency sub-band grid of the discrete wavelet transform and to complete the weighted fusion, and to generate a fused feature map through inverse discrete wavelet transform;

[0113] The super-resolution image generation module 404 is used to perform convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and to generate a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image.

[0114] In one embodiment, the wavelet domain trust fusion module 403 is further configured to: calculate the normalized amplitude difference between the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients, and calculate the amplitude difference gating weight through a first learnable gating function; calculate the local phase difference between the two and perform angle-based circling and normalization processing, and calculate the phase consistency gating weight through a second learnable gating function; calculate the trust weight based on the amplitude difference gating weight, the phase consistency gating weight, and the basic weight; merge the six-directional trust weights into three-directional trust weights according to the directional angle symmetry pairing relationship, and align them to the discrete wavelet transform high-frequency subband grid through spatial interpolation; apply complementary hierarchical weights to the trust weights of different decomposition levels according to the hierarchical emphasis coefficient; and fuse the low-frequency subband according to the thermal infrared dominance principle.

[0115] In one embodiment, the super-resolution image generation module 404 is further configured to: perform physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image; and perform physical consistency edge thinning processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image.

[0116] Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0117] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0118] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal infrared image super-resolution reconstruction method based on wavelet domain trust gating, characterized in that, The method includes: acquiring a low-resolution thermal infrared image and a corresponding high-resolution visible light image; using the low-resolution thermal infrared image as a reference, performing spatial registration processing on the high-resolution visible light image to obtain a registered visible light image; performing feature extraction processing on the low-resolution thermal infrared image and the registered visible light image respectively to obtain thermal infrared feature maps and visible light feature maps; performing dual-tree complex wavelet transform on the thermal infrared feature maps and the visible light feature maps respectively to obtain thermal infrared complex wavelet coefficients and visible light complex wavelet coefficients; the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients are both organized according to decomposition levels and directional sub-bands, and each coefficient contains local amplitude and local phase; based on the local phase difference and local amplitude difference of the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients at corresponding levels, corresponding directions, and corresponding spatial positions. The process involves calculating trust weights hierarchically, directionally, and spatially; performing discrete wavelet transforms on the thermal infrared feature map and the visible light feature map respectively to obtain their respective low-frequency subbands and multi-directional high-frequency subbands; mapping the trust weights to the high-frequency subband grid of the discrete wavelet transform, and weighted fusing the high-frequency subbands of the visible light feature map and the thermal infrared feature map based on the mapped trust weights to obtain a fused high-frequency subband; fusing the low-frequency subbands of the thermal infrared feature map and the visible light feature map to obtain a fused low-frequency subband; performing inverse discrete wavelet transforms on the fused low-frequency subband and the fused high-frequency subband to generate a fused feature map; performing convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and generating a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image.

2. The method according to claim 1, characterized in that, The step of calculating trust weights for each level, direction, and spatial location based on the local phase and amplitude differences of the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients at corresponding levels, directions, and spatial locations includes: calculating the absolute difference between the local amplitudes of the thermal infrared complex wavelet coefficients and the visible light complex wavelet coefficients, and normalizing the absolute difference to obtain a normalized amplitude difference; and calculating the amplitude difference gating weights based on the normalized amplitude difference using a first learnable gating function: M_H = ( z_m / (1+|z_m|) + 1 ) / 2 , where z_m = β_m·(ΔM_norm -ρ_m) where M_H is the amplitude difference gating weight, ΔM_norm is the normalized amplitude difference, and β_m and ρ_m are learnable parameters; calculate the absolute phase difference between the local phase of the thermal infrared complex wavelet coefficients and the local phase of the visible light complex wavelet coefficients, and perform angle wrapping and normalization processing on the absolute phase difference to obtain a local phase consistency index; based on the local phase consistency index, calculate the phase consistency gating weight through the second learnable gating function: P = ( z_p / (1+|z_p|) + 1 ) / 2 , where z_p = β_p·( (1 − |Δφ| / π) − ρ_p ) where P is the phase consistency gating weight, Δφ is the local phase difference after angle wrapping processing, and β_p and ρ_p are learnable parameters; based on the amplitude difference gating weight, the phase consistency gating weight, and the base weight, calculate the trust weight using the following formula: w = clamp( w_0 + k·P·M_H , w_0 , w_max ) where w is the trust weight, w_0 is the base weight greater than zero, k is the adjustment coefficient, clamp is the constraint function, and w_max is the upper limit of the weight; the base weight w_0 ensures that the visible light contribution at any level, any direction, and any spatial location is not completely turned off.

3. The method according to claim 1 or 2, characterized in that, The step of mapping the trust weights to the high-frequency subband grid of the discrete wavelet transform includes: pairing and merging the six directional subbands of the dual-tree complex wavelet transform into three sets of directional trust weights that correspond one-to-one with the three high-frequency directional subbands of the discrete wavelet transform, according to the symmetrical pairing relationship of the directional angles; wherein, the near-horizontal directional pairs are the first set, the near-vertical directional pairs are the second set, and the diagonal directional pairs are the third set, and the trust weights in each set are merged by taking the average value; when the spatial grid size of the merged trust weights is inconsistent with the spatial grid size of the high-frequency subband of the corresponding level of the discrete wavelet transform, the merged trust weights are aligned to the spatial grid of the high-frequency subband by spatial interpolation.

4. The method according to claim 1, characterized in that, The method further includes hierarchical emphasis processing: setting learnable hierarchical emphasis parameters for each feature channel, and converting the hierarchical emphasis parameters into hierarchical emphasis coefficients through bounded mapping; Based on the hierarchical emphasis coefficient, complementary hierarchical weights are applied to the trust weights of different decomposition levels, so that different wavelet domain trust fusion units are differentiated into scale divisions emphasizing different decomposition levels during training; wherein, the hierarchical weights are provided with a lower limit greater than zero to ensure that no decomposition level is completely turned off.

5. The method according to claim 1, characterized in that, The process of fusing the low-frequency sub-bands of the thermal infrared feature map and the visible light feature map to obtain a fused low-frequency sub-band includes: calculating the absolute difference between the low-frequency sub-bands of the thermal infrared feature map and the low-frequency sub-bands of the visible light feature map, and performing normalization processing to obtain a normalized low-frequency difference; calculating the low-frequency thermal infrared weight based on the normalized low-frequency difference according to the thermal infrared dominance principle: the smaller the normalized low-frequency difference, the closer the low-frequency thermal infrared weight is to a preset upper limit; the preset upper limit is close to 1; and performing weighted fusion of the low-frequency sub-bands of the thermal infrared feature map and the low-frequency sub-bands of the visible light feature map based on the low-frequency thermal infrared weight, so that the fused low-frequency sub-band is dominated by the thermal infrared low-frequency component, in order to maintain the overall thermal radiation distribution characteristics of the thermal infrared image.

6. The method according to claim 2, characterized in that, The method further includes cross-directional coherence correction processing: neighborhood aggregation is performed on the phase consistency degree on adjacent directional sub-bands within the same decomposition level to obtain a cross-directional coherence index; the trust weight is corrected based on the cross-directional coherence index, so that spatial positions where phase consistency in adjacent directions corroborates each other obtain a relatively higher trust weight, and spatial positions where phase consistency is isolated and anomalous obtain a relatively lower trust weight.

7. The method according to claim 1, characterized in that, The step of generating a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image includes: performing physical consistency high-frequency enhancement processing on the initial super-resolution thermal infrared image to obtain a high-frequency enhanced image; wherein, the physical consistency high-frequency enhancement processing includes: relatively enhancing the thermal infrared amplitude spectrum in the high-frequency region based on the normalized visible light amplitude spectrum, keeping the thermal infrared phase spectrum and the amplitude spectrum in the low-frequency region unchanged, and mixing it with the initial super-resolution thermal infrared image through a decaying mixing coefficient; performing physical consistency edge refinement processing on the high-frequency enhanced image to generate a super-resolution thermal infrared image; wherein, the physical consistency edge refinement processing includes: extracting edge features and edge confidence masks from the registered visible light image, generating a residual map and applying zero-mean constraints and amplitude limiting constraints, weighting it with the edge confidence mask, and mixing it with the high-frequency enhanced image through a decaying mixing coefficient.

8. A multimodal infrared image super-resolution reconstruction system based on wavelet domain trust gating, characterized in that, The system includes: an image registration module for acquiring a low-resolution thermal infrared image and a corresponding high-resolution visible light image; and spatially registering the high-resolution visible light image with the low-resolution thermal infrared image as a reference to obtain a registered visible light image; a feature extraction module for performing feature extraction on the low-resolution thermal infrared image and the registered visible light image respectively to obtain a thermal infrared feature map and a visible light feature map; a wavelet domain trust fusion module for performing dual-tree complex wavelet transform on the thermal infrared feature map and the visible light feature map respectively, and calculating trust weights based on local phase differences and local amplitude differences at corresponding levels, corresponding directions, and corresponding spatial positions; performing discrete wavelet transform on the thermal infrared feature map and the visible light feature map respectively, mapping the trust weights to the high-frequency subband grid of the discrete wavelet transform and completing weighted fusion, and generating a fused feature map through inverse discrete wavelet transform; and a super-resolution image generation module for performing convolution processing on the fused feature map to generate an initial super-resolution thermal infrared image; and generating a super-resolution thermal infrared image based on the initial super-resolution thermal infrared image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images