Image fusion method and system based on cross-modal attention explicit task guidance

Abstract

This invention discloses an image fusion method and system based on cross-modal attention explicit task guidance. The method includes: normalizing and extracting features from infrared and visible light images to obtain infrared and visible light image features; performing feature decomposition on the infrared and visible light image features using discrete wavelet transform to obtain low-frequency and high-frequency components; generating an explicit task guidance signal and injecting it into the high and low-frequency components through a cross-modal attention mechanism for target perception feature enhancement, resulting in enhanced high and low-frequency components; and classifying and adaptively fusing the enhanced high and low-frequency components to obtain a target-perception fused image. This invention effectively preserves infrared thermal energy distribution and visible light details, improving the visual fusion effect of the images. As an image fusion method and system based on cross-modal attention explicit task guidance, this invention can be widely applied in the field of cross-modal image fusion technology.

Description

Technical Field

[0001] This invention relates to the field of cross-modal image fusion technology, and in particular to an image fusion method and system based on cross-modal attention explicit task guidance. Background Technology

[0002] With the rapid development of multi-sensor imaging technology, infrared and visible light imaging have become core tools in complex environment perception and intelligent vision systems. Infrared imaging captures the difference in thermal radiation between the target and the background, and can still image stably under conditions such as nighttime, low light, smoke obscuration, and complex weather, thus having significant advantages in target detection and contour perception. However, due to limitations in imaging mechanisms and hardware, infrared images typically suffer from low resolution, lack of texture and structural details, and blurred background layers, making it difficult to meet the needs of refined perception. In contrast, visible light imaging can provide rich texture, color, and geometric structure information, and has high visual readability in daylight or under sufficient lighting conditions, but its imaging quality is heavily dependent on ambient light and is prone to information degradation under low light, backlight, or complex weather conditions. These complementary characteristics mean that a single modality cannot comprehensively and stably characterize the features of a real scene, thus easily leading to the risk of missed detections or false detections in downstream tasks such as target detection and recognition.

[0003] To fully integrate complementary information across spectral bands, Infrared and Visible Image Fusion (IVIF) technology has emerged. Its goal is to effectively fuse salient target information in infrared images with rich texture and structural information in visible images to generate a single image with high signal-to-noise ratio, clear details, and prominent targets. This provides more reliable visual input for mission-driven applications such as security monitoring, drone inspection, autonomous driving, and military reconnaissance.

[0004] From a methodological evolution perspective, IVIF technology has gradually developed from traditional model-driven methods to deep learning methods. Early research mainly relied on ideas such as multi-scale transformation, sparse representation, low-rank decomposition, and saliency modeling, achieving cross-modal information integration through hand-designed fusion rules. Although these methods possessed interpretability to some extent, their feature representation capabilities were limited, making them difficult to adapt to complex and ever-changing real-world scenarios. With the development of deep learning, research focus gradually shifted to data-driven fusion models, including architectures such as autoencoders (AEs), convolutional neural networks (CNNs), generative adversarial networks (GANs), and Transformers. These methods, by introducing deep feature learning mechanisms, significantly improved the expressive power and fusion flexibility of cross-modal information.

[0005] However, existing deep learning methods still have inherent limitations that cannot be ignored. While GAN-like methods can enhance the overall contrast and visual expressiveness of fused images, their adversarial training process is highly unstable, easily introducing spectral distortion or artifacts, and the training process is highly sensitive to hyperparameter settings. Transformer models, with their self-attention mechanism, possess the ability to model global dependencies and excel in multimodal feature interactions, but their computational and storage costs increase quadratically with input resolution, severely limiting their practical deployment in high-resolution scenes and resource-constrained platforms.

[0006] In recent years, the IVIF research paradigm has been gradually shifting from subjective quality optimization centered on "human visual preferences" to objective performance optimization aimed at "machine task-driven" performance. Some studies have attempted to directly input fused images into downstream object detection or recognition networks and use task loss to perform end-to-end joint optimization of the fusion network. This strategy can effectively improve detection performance in scenes with dense or salient targets. However, when there are no obvious targets in the input scene, the detection loss often produces unstable or even misleading gradient signals, thereby interfering with the cross-modal feature interaction process and causing degradation in texture fidelity and structural consistency of the fusion results. In addition, the trade-off between fusion quality and task performance in existing methods usually relies on human experience to adjust the loss weights, which is not only costly to tune but also difficult to adapt to different scenes and data distributions, severely limiting the model's generalization ability and practicality. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention aims to provide an image fusion method and system based on cross-modal attention explicit task guidance, which can effectively preserve infrared thermal energy distribution and visible light details, thereby improving the visual fusion effect of images.

[0008] The first technical solution adopted in this invention is: an image fusion method based on cross-modal attention explicit task guidance, comprising the following steps: Infrared and visible light images are normalized and then input into a state-space model and a convolutional neural network model, respectively, for feature extraction to obtain infrared image features and visible light image features. The infrared and visible light image features are decomposed using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. An explicit task guidance signal is generated, and the explicit task guidance signal is injected into the low-frequency component containing the main contour and the high-frequency component containing the texture edge through a cross-modal attention mechanism to perform feature enhancement processing for target perception, resulting in enhanced high and low frequency components. The enhanced high and low frequency components are classified and adaptively fused, and the fused features are reconstructed through wavelet inverse transform to obtain the target perception fused image.

[0009] Furthermore, the step of normalizing the infrared and visible light images and inputting them into the state-space model and convolutional neural network model respectively for feature extraction to obtain infrared image features and visible light image features specifically includes: Two-dimensional infrared images and two-dimensional visible light images are acquired and converted to obtain one-dimensional vector sequences of infrared images and visible light images. The infrared image feature map and the visible light feature map are obtained by sequentially performing image mapping and image flattening processing on the one-dimensional vector sequence of the infrared image and the one-dimensional vector sequence of the visible light image through the overlapping block method. Based on the state-space model, the SS2D module is introduced to perform four-way feature scanning and extraction on the infrared feature map and the visible light feature map to obtain the infrared global feature and the visible light global feature. By performing local feature constraint processing on the global features of infrared and visible light using a convolutional neural network model, infrared image features and visible light image features are obtained.

[0010] Furthermore, the step of performing feature decomposition processing on infrared image features and visible light image features using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges specifically includes: Define the discrete wavelet basis function and calibration function and perform inner product calculation to obtain the detail coefficients and approximation coefficients, and construct the decomposition expression of the wavelet basis signal at a preset scale; By introducing low-frequency adaptive analysis vectors and high-frequency adaptive analysis vectors, and combining them with the decomposition expression of wavelet basis signals at a preset scale, four sets of convolution operators are constructed. The four sets of convolution operators include low-frequency, horizontal high-frequency, vertical high-frequency, and diagonal high-frequency convolution operators. Four sets of convolution operators are used to convolve and downsample the infrared image features and visible light image features in the row and column directions, respectively, to obtain the first-level low-frequency approximation component, the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component and the first-level high-frequency diagonal detail component. The DWT filter bank is used to perform secondary decomposition on the first-level low-frequency approximation component, the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, and the first-level high-frequency diagonal detail component to obtain the second-level low-frequency approximation component, the second-level high-frequency horizontal detail component, the second-level high-frequency vertical detail component, and the second-level high-frequency diagonal detail component. The second-order low-frequency approximation component is injected into the global semantics of the target detection network to construct a low-frequency component containing the main contour; By employing a spatial activity adaptive enhancement strategy, local variance calculations are performed on the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, the first-level high-frequency diagonal detail component, the second-level high-frequency horizontal detail component, the second-level high-frequency vertical detail component, and the second-level high-frequency diagonal detail component to obtain high-frequency components that include texture edges.

[0011] Furthermore, the step of generating an explicit task guidance signal and injecting it into the low-frequency component containing the main contour and the high-frequency component containing the texture edges through a cross-modal attention mechanism for target perception feature enhancement processing to obtain enhanced high and low frequency components specifically includes: The detected image is input into a pre-trained target detection network, the intermediate layer feature maps of its deep backbone network are extracted, and an explicit task guidance signal is generated after dimension alignment. Construct parallel cross-dimensional attention modules, using low-frequency components containing the main contour and high-frequency components containing texture edges as query terms, and explicit task guidance signals as key and value terms. By learning the weight matrix, linear projections are performed on the query item, key item, and value item respectively, and the correlation between each pixel of the wavelet component and the inspection prompt signal is calculated, target perception attention is generated. By using a residual injection structure, target perception attention is injected into the low-frequency component containing the main contour and the high-frequency component containing the texture edge for target perception feature enhancement processing, resulting in enhanced high and low frequency components.

[0012] Furthermore, the step of classifying and adaptively fusing the enhanced high and low frequency components, and reconstructing the fused features through wavelet inverse transform to obtain the target-aware fused image, specifically includes: An adaptive strategy based on energy weights is used to fuse the enhanced low-frequency components to obtain the fused low-frequency components. Based on the spatial activity selection strategy, the enhanced high-frequency components are fused to obtain the fused high-frequency components. The low-frequency component and the high-frequency component after fusion are reconstructed by multi-scale spatial reconstruction using the inverse discrete wavelet transform method to obtain the reconstructed feature components. The reconstructed feature components are input into a decoding network containing residual connections, and mean information from infrared and visible light images is introduced through long-range skip connections to perform global contrast restoration and noise suppression, resulting in a target perception fusion image.

[0013] Furthermore, the expression for the energy-weight-based adaptive strategy is as follows: In the above formula, This represents the low-frequency component after fusion. Represents the low-frequency component of infrared radiation. This represents the low-frequency component of visible light. , This represents the weighting coefficient.

[0014] Furthermore, the expression for the spatial activity-based selection strategy is as follows: In the above formula, This represents the high-frequency components after fusion. Represents the low-frequency component of infrared radiation. This represents the low-frequency component of visible light. This indicates taking the maximum value element by element.

[0015] Furthermore, the expression for the multi-objective optimization function of the target-aware fusion image is as follows: In the above formula, Represents a multi-objective optimization function. Indicates strength loss. Represents gradient loss, Represents structural similarity loss. Indicates detection loss, , , , This represents the weighting parameter.

[0016] The second technical solution adopted in this invention is: an image fusion system based on cross-modal attention explicit task guidance, comprising: The first module is used to normalize infrared and visible light images and input them into the state space model and convolutional neural network model respectively for feature extraction to obtain infrared image features and visible light image features. The second module is used to perform feature decomposition processing on infrared image features and visible light image features using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. The third module is used to generate explicit task guidance signals and inject the explicit task guidance signals into the low-frequency components containing the main contours and the high-frequency components containing texture edges through a cross-modal attention mechanism to perform feature enhancement processing for target perception, resulting in enhanced high and low frequency components. The fourth module is used to classify and adaptively fuse the enhanced high and low frequency components, and reconstruct the fusion features through wavelet inverse transform to obtain the target-aware fusion image.

[0017] The beneficial effects of the method and system of this invention are as follows: This invention normalizes infrared and visible light images and inputs them into a state-space model and a convolutional neural network model, respectively, for feature extraction, obtaining infrared image features and visible light image features. The introduction of the Mamba model significantly improves the modeling ability for large targets and overcomes the feature loss caused by the limited receptive field of traditional convolutional networks. Furthermore, the discrete wavelet transform method is used to perform feature decomposition on the infrared and visible light image features, obtaining low-frequency components containing the main contours and high-frequency components containing texture edges. The low-frequency guided space can forcibly enhance the pixel intensity of the target area, making the infrared thermal target have higher contrast significance in the fused image. The high-frequency guided space calculates the local variance of each pixel to enhance the detection... The target edge is targeted and reinforced while high-frequency noise in the background region is suppressed. An explicit task guidance signal is then generated and injected into the low-frequency component containing the main contour and the high-frequency component containing textured edges via a cross-modal attention mechanism for target perception feature enhancement, resulting in enhanced high and low frequency components. Generating the explicit task guidance signal solves the problem of interference with fusion quality caused by the detection task in the absence of salient target samples. Finally, the enhanced high and low frequency components are classified and adaptively fused, and the fusion features are reconstructed using inverse wavelet transform to obtain a target-perceived fused image. Wavelet transform is used to differentiate target saliency and background texture in the frequency domain, effectively preserving infrared thermal energy distribution and visible light details, thus improving the visual fusion effect of the image. Attached Figure Description

[0018] Figure 1 This is a flowchart of the steps of the image fusion method based on cross-modal attention explicit task guidance of the present invention; Figure 2 This is a block diagram of the image fusion system based on cross-modal attention explicit task guidance of the present invention; Figure 3 This is a schematic diagram of image fusion provided by the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0020] Reference Figure 1 This invention provides an image fusion method based on cross-modal attention explicit task guidance, which includes the following steps: S100. Normalize the infrared image and the visible light image and input them into the state space model and the convolutional neural network model respectively for feature extraction to obtain the infrared image features and the visible light image features. Specifically, two-dimensional infrared and two-dimensional visible light images are acquired and converted to obtain one-dimensional vector sequences of infrared and visible light images. The one-dimensional vector sequences of infrared and visible light images are then processed sequentially using an overlapping block method to perform image mapping and image flattening, resulting in infrared and visible light feature maps. Based on a state-space model, an SS2D module is introduced to perform four-directional feature scanning extraction on the infrared and visible light feature maps, yielding global infrared and visible light features. Finally, a convolutional neural network model is used to perform local feature constraint processing on the global infrared and visible light features, resulting in infrared image features and visible light image features.

[0021] In this embodiment, normalized infrared and visible light images are mapped to a high-dimensional feature space through an overlapping block embedding module with shared weights, and then fed into a non-uniform hybrid encoder composed of a state-space model (Mamba) and a convolutional neural network (CNN). Within this encoder, multiple front-end MambaBlock modules capture the long-range spatial dependencies of the image through a selective state-space scanning mechanism, achieving macroscopic modeling of the target's global contour; the rear-end modules further refine edge and texture details by leveraging the local receptive field advantage of the C2f structure. This hybrid architecture, by deeply integrating Mamba's linear time complexity long-range modeling capability with the local inductive bias characteristics of CNNs, effectively achieves simultaneous representation of global semantics and local details during the feature extraction stage, laying a data foundation for subsequent accurate feature decoupling and task guidance.

[0022] More specifically, first, the two-dimensional image Convert to a one-dimensional vector sequence; utilize overlapping block technology, with a stride of 4 and a convolution kernel size of [missing value]. The convolution operator maps the image to a scale of The number of channels is The feature map is flattened in the spatial dimension to obtain a feature map of length . sequence , as input to a Mamba block.

[0023] Traditional Mamba models are causal and can only handle unidirectional sequences. To enable the model to perceive the spatial topological relationships of two-dimensional images, this embodiment uses the SS2D (2D-Selective Scan) module, which expands the input sequence symmetrically along four directions: from top left to bottom right (forward scan); from bottom right to top left (backward scan); transposed scan (from bottom left to top right); and reverse transposed scan (from top right to bottom left). The sequence in each direction is independently fed into a state-space model (SSM) for feature extraction, ensuring that each pixel receives global information from all four quadrants of the image, thus establishing complete long-range spatial dependencies; the feature flow in each scan direction follows the state-space definition of a linear time-invariant system. The continuous system is discretized using a zero-order hold (ZOH) and a time step parameter is introduced. , will continuous parameters Convert to discrete parameters : The state is then updated using the following difference equation: parameter It's not fixed, but rather determined by input. The linear projection is dynamically generated. This means that the model can adaptively adjust the weights of information that is "forgotten" or "retained" based on the image content, such as prominent thermal targets in infrared images.

[0024] After the Mamba block captures the global definition, by combining the local inductive bias of the CNN: the output of the Mamba block passes through a... The depthwise separable convolutional layer (DWConv) is used. Convolutional operations provide the local spatial constraints missing in Mamba, which can correct the weakening of adjacent pixel relationships caused by serialization. In the first two stages of the encoder (low-resolution features), Mamba blocks are placed in an important manner to widen the receptive field; in the last two stages (high-resolution details), C2f convolutional modules are added to refine the edge information of the target by utilizing their cross-layer connectivity.

[0025] By modeling the linear time complexity using Mamba, this invention achieves a memory footprint that increases linearly with resolution when processing high-resolution infrared and visible light images, effectively addressing the issue of high computational overhead in fusion tasks using the Transformer architecture. Simultaneously, the SS2D mechanism ensures that the macroscopic contours of infrared thermal targets and the global context of the visible light background are deeply integrated during the feature extraction stage.

[0026] S200. The infrared image features and visible light image features are decomposed by discrete wavelet transform to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. Specifically, discrete wavelet basis functions and calibration functions are defined and their inner products are calculated to obtain detail coefficients and approximation coefficients, thus constructing a decomposition expression for the wavelet basis signal at a preset scale. Low-frequency and high-frequency adaptive analysis vectors are introduced, and combined with the decomposition expression of the wavelet basis signal at the preset scale, four sets of convolution operators are constructed. These four sets of convolution operators include low-frequency, horizontal high-frequency, vertical high-frequency, and diagonal high-frequency convolution operators. The infrared image features and visible light image features are convolved and downsampled in the row and column directions using these four sets of convolution operators, respectively, to obtain a first-level low-frequency approximation component, a first-level high-frequency horizontal detail component, a first-level high-frequency vertical detail component, and a first-level high-frequency diagonal detail component. The first-level low-frequency approximation component is then processed using a DWT filter bank. The first-level high-frequency approximation component, the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, and the first-level high-frequency diagonal detail component are decomposed into second-level low-frequency approximation components, second-level high-frequency horizontal detail components, second-level high-frequency vertical detail components, and second-level high-frequency diagonal detail components. The second-level low-frequency approximation component is injected with the global semantics of the target detection network to construct a low-frequency component containing the main contour. Through a spatial activity adaptive enhancement strategy, the local variance of the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, the first-level high-frequency diagonal detail component, the second-level high-frequency horizontal detail component, the second-level high-frequency vertical detail component, and the second-level high-frequency diagonal detail component is calculated for each pixel to obtain a high-frequency component containing texture edges.

[0027] In this embodiment, the Discrete Wavelet Transform (DWT) decomposition step specifically involves using the Haar wavelet basis function as a filtering operator to perform a two-level spatial downsampling decomposition on the baseline features. In each decomposition stage, low-frequency approximate components representing the main energy and macroscopic contours of the image are extracted through low-pass filtering in the horizontal and vertical directions, and three high-frequency detail components representing the image edges and texture details are extracted through high-pass filtering in the horizontal, vertical, and diagonal directions. The decomposed components are halved in spatial resolution and rearranged in the channel dimension to achieve decoupled representation of features in the spatial and frequency domains.

[0028] More specifically, the reference features of the hybrid encoder output are obtained by employing two-dimensional discrete wavelet transform (2D-DWT). Non-destructive decomposition is performed. Because Haar wavelets possess orthogonality and support, they can achieve spatial-frequency localization of features in linear time and have extremely high computational efficiency in discretization. In constructing the wavelet transform, the case of a one-dimensional signal is considered first. The discrete wavelet transform is defined as... and calibration function and the signal at scale The decomposition is as follows: in Given a discrete input signal; It is a scaling factor; Time factor and calibration function , Indicates detail coefficients. Representing the approximation coefficients. Introducing an adaptive analysis vector. and These correspond to low-frequency and high-frequency filters, respectively. The decomposition of the wavelet basis signal can be viewed as a recursive convolution with a specific filter: For any input image The two-dimensional image can be decomposed as follows: in , , and These represent the low-frequency, horizontal high-frequency, vertical high-frequency, and diagonal high-frequency convolution operators achieved through grouped convolution, respectively.

[0029] Specific steps and functions: Define a low-pass filter and high-pass filter This is equivalent to smoothing (summing) and extracting differences (difference) on the feature map.

[0030] Feature tensor First, proceed in the horizontal direction. and Convolution and downsampling are performed, and then the operation is repeated in the column direction. This process consists of four components: (Low-frequency approximation component): It represents the tone of the source image, carrying the intensity distribution of the infrared thermal target and the background outline of the visible light scene.

[0031] (Horizontal details): Extract burst features from vertical edges.

[0032] (Vertical details): Extract abrupt changes in horizontal edges.

[0033] (Diagonal details): Extract texture and high-frequency noise in the diagonal direction.

[0034] To further strip away the deep semantic energy, this implementation... The components are recursively decomposed into two levels, and then... As input, it passes through the DWT filter bank again to obtain deeper low-frequency components. and second-order high-frequency components { } The spatial resolution of the component is the same as that of the original image. This allows the subsequent "explicit task guidance" to be precisely applied to the centroid region of the target, avoiding interference from detail noise on the injection of the detection signal.

[0035] Low-frequency guidance space: targeting Components, injected into the global semantics of the target detection network (i.e., "explicit spatial cues"), forcefully enhance the pixel intensity of the target region, making infrared thermal targets have higher contrast saliency in the fused image. High-frequency guided space: targeting { For each component, a spatial activity adaptive enhancement strategy is adopted. By calculating the local variance of each pixel, the detected target edges are targeted and enhanced, while high-frequency noise in the background region is suppressed.

[0036] S300: Generate explicit task guidance signals, and inject the explicit task guidance signals into the low-frequency components containing the main contours and the high-frequency components containing texture edges through a cross-modal attention mechanism to perform feature enhancement processing for target perception, thereby obtaining enhanced high and low frequency components. Specifically, the detected image is input into a pre-trained target detection network, and the intermediate layer feature maps of its deep backbone network are extracted and processed by dimension alignment to generate an explicit task guidance signal. A parallel cross-dimensional attention module is constructed, using the low-frequency component containing the main contour and the high-frequency component containing texture edges as query terms, and the explicit task guidance signal as key and value terms. The query terms, key terms, and value terms are linearly projected onto the learned weight matrix, and the correlation between each pixel of the wavelet component and the inspection prompt signal is calculated to generate target perception attention. Through a residual injection structure, the target perception attention is injected into the low-frequency component containing the main contour and the high-frequency component containing texture edges for target perception feature enhancement processing, resulting in enhanced high and low frequency components.

[0037] In this embodiment, the detected image is input into a pre-trained target detection network, and the intermediate layer feature maps of its deep backbone network are extracted and generated as explicit task guidance signals after dimensional alignment processing. A parallel cross-dimensional attention module is constructed, using the low-frequency and high-frequency components after wavelet decomposition as queries, and the explicit task guidance signals as keys and values. By calculating the spatial correlation matrix between the queries and keys, the category and location semantic information in the target detection task are directionally weighted to the wavelet components at each level, and the enhanced feature components that fuse task-aware information are output.

[0038] More specifically, a pre-trained object detection network (YOLOv8) is used as the semantic source. To obtain cue signals with high abstraction and spatial saliency, feature layers at the ends of the network backbone are extracted. Feature maps from layers P4 or P5, which contain rich semantic information, are selected and denoted as... These features are trained on detection data and have activation responses to target locations (such as pedestrians and vehicles). During the training of the fusion network, all parameters of the detection network are frozen, ensuring that the task signal is a stable "explicit cue" rather than a "stochastic gradient" that fluctuates with the fusion network.

[0039] Due to detection features The size is usually smaller than that of the wavelet component. Spatial and channel alignment is required: spatial alignment uses bilinear interpolation to... Upsampled to the wavelet component to be enhanced (e.g.) The same resolution ); Channel alignment via a The convolutional layer will detect the number of channels. Channel dimension mapped to the converged network Obtain the aligned spatial cue signal .

[0040] The cross-modal attention module is constructed by explicitly "injecting" the detected semantics into the wavelet domain through an attention mechanism. The query (Q) is derived from wavelet component features in the image fusion branch, specifically low-frequency guided spatial features and high-frequency guided spatial features, denoted as... The key (Key, K) and value (Value, V) are derived from the cue signals of the detection branch. It carries the target location information and is released using a learned weight matrix. Perform linear projection: Calculate the correlation between each pixel element of the wavelet component and the inspection cue signal to generate target-aware attention: in, `x` is a scaling factor used to prevent gradient vanishing due to excessively large dot products. The resulting attention output includes features weighted by the target cues. To preserve the integrity of the original image and enhance target saliency, a residual injection structure is employed. This step maps the "objectness" location identified by the detection network to wavelet components. For example, in the low-frequency infrared component ( In this operation, the response value of the pedestrian or vehicle area is explicitly enhanced, giving it higher contrast after visual reconstruction.

[0041] This explicit guidance method has significant technical advantages compared to the traditional "end-to-end loss feedback" method: 1) In traditional gradient backpropagation methods, if the input image does not contain the target, the detection loss will force the network to change its weights in order to find the non-existent target, which will lead to artifacts.

[0042] 2) In this embodiment, if there are no significant targets in the scene, the cue signals extracted by the detection network are used. The activation value is extremely low, expressed as: when , At this point, the enhancement formula degenerates into In other words, in scenarios without a target, the fusion network retains only the original physical fusion features and is not misled by erroneous gradients.

[0043] S400: The enhanced high and low frequency components are classified and adaptively fused, and the fusion features are reconstructed through wavelet inverse transform to obtain the target perception fusion image.

[0044] Specifically, based on an adaptive strategy using energy weights, the enhanced low-frequency components are fused to obtain the fused low-frequency components; based on a spatial activity selection strategy, the enhanced high-frequency components are fused to obtain the fused high-frequency components; the fused low-frequency components and the fused high-frequency components are then subjected to multi-scale spatial reconstruction processing using the discrete wavelet inverse transform method to obtain the reconstructed feature components; the reconstructed feature components are input into a decoding network containing residual connections, and the mean information of the infrared and visible light images is introduced through long-range skip connections to perform global contrast restoration and noise suppression, resulting in a target perception fused image.

[0045] In this embodiment, adaptive weighted aggregation is used for the enhanced infrared and visible low-frequency components to preserve the energy distribution of the source image; a spatial activity-based selective strategy is used for the enhanced high-frequency components to extract fine textures and edges; the fused low-frequency components and high-frequency components at all levels are used as inputs, and multi-scale spatial reconstruction is performed using inverse discrete wavelet transform (IDWT) to restore the frequency domain features to the original image space; the reconstructed features are input into a decoding network containing residual connections, and the mean information of the source image is introduced through long-range skip connections to perform global contrast restoration and noise suppression on the fusion result, generating the final target-aware fused image.

[0046] More specifically, before performing the inverse transformation, the infrared components that have undergone mission-guided enhancement need to be... With visible light component To integrate.

[0047] Low-frequency component (LL) fusion: An adaptive strategy based on energy weights is employed to preserve the intensity of infrared thermal targets. Among them, weight Calculated dynamically based on the energy distribution in the local area.

[0048] High-frequency (LH, HL, HH) fusion: Employing a spatial activity-based selection strategy to preserve visible light texture edges. Subsequently, the multi-scale frequency domain components are restored step by step using the discrete wavelet inverse transform. IDWT upsamples the sub-bands and applies a reconstruction filter (low-pass filter). and Qualcomm This is achieved by ). Let the subbands after the second-order decomposition be { }, Level 1 reconstruction yields : Repeat the above process, using Preliminary fusion feature map with original resolution obtained by reconstructing from the first-level high-frequency subband. .

[0049] Since wavelet transform is a linear operator, direct reconstruction may not completely eliminate phase distortion or block effects introduced during the enhancement process. This embodiment introduces a residual decoder for nonlinear optimization. The decoder 7531 consists of several residual blocks. Each block contains two... Convolutional layers and skip connections.

[0050] Through residual learning, the network only needs to learn the "residual information" between the reconstructed features and the ideal fused image, which greatly reduces the training difficulty and effectively compensates for texture details.

[0051] To ensure image fusion To avoid brightness collapse or overexposure, this embodiment incorporates the global mean Info of the source image for contrast adjustment. The pixel mean of the infrared and visible light source images is calculated as follows: The decoder output is weighted and fused with the mean information, and an activation function is applied to limit the range: in This represents element-level bias addition. This step ensures that even with significant enhancement of the task-driven thermal target, the background area still maintains a natural brightness and hue consistent with human visual perception.

[0052] Furthermore, a joint loss function is established that couples image quality constraints and task accuracy constraints. The image fusion loss term includes intensity mean square error loss, structural similarity loss, and correlation loss, while the target detection loss term includes classification loss, localization loss, and confidence loss. Weighting coefficients to be optimized are set for each loss term, and these weighting coefficients are mapped to individual gene codes in an evolutionary algorithm. A genetic evolutionary algorithm is adopted, using the mean accuracy (mAP) on the validation set as the fitness function. Through selection, crossover, and mutation operations, the weighting coefficients are evolved over multiple generations to automatically search for the loss weight combination that optimizes the performance of the downstream detection task.

[0053] More specifically, a multi-objective optimization function was constructed to balance the visual fidelity of the image with the accuracy of object detection. The total loss function... Image fusion sub-loss With target detection sub-loss Weighted composition. Total loss formula: Strength loss To maintain brightness consistency between the fused image and the source image, the following methods are typically used. Norm.

[0054] Gradient loss Constraint-fused images extract texture details from the source image. .

[0055] Structural similarity loss Maintain the stability of the image's topological structure.

[0056] Detection loss : Including the regression loss of the prediction box (such as CIoU) and class probability loss .

[0057] Finding the optimal weight vector This allows the model to achieve optimal performance for a specific task.

[0058] To achieve automated hyperparameter search, this embodiment introduces a genetic algorithm (GA) or an evolutionary strategy. The weight vector... Considered as an individual's "chromosome". Each gene locus In the real number space Encoding is performed internally. N sets of initial weight individuals are randomly generated to form the initial population. .

[0059] The fitness function is the core driver of evolution. The fitness value is directly used as the performance metric of the machine learning task. Evaluation process: 1) Take a set of weight parameters from the population. .

[0060] 2) Use this weight configuration to perform short-term, rapid training of the fusion model (e.g., training for 5 epochs).

[0061] 3) Run the pre-trained target detector on the validation set and output the detection results.

[0062] Fitness calculation formula: in, It is the main evaluation indicator (mean precision). For balance coefficient, This is used to ensure that image quality does not deteriorate. The main focus is on maximizing detection accuracy on the validation set.

[0063] By simulating the natural selection process, the weight parameters are optimized generation by generation. An elite retention strategy is adopted, selecting the top 50% of individuals by fitness level to enter the mating pool, and pairing two parent individuals... and Perform arithmetic crossover to generate new offspring individuals: in is a random crossover factor.

[0064] In order to escape local optima, a set mutation probability is used. (e.g., 0.1) Apply a Gaussian perturbation to the gene locus: The specific automation process in this embodiment is as follows: 1) Set the maximum number of generations of evolution. (e.g., 30 generations) and population size.

[0065] 2) In each generation, the weights of each group are assigned to parallel training tasks to obtain the corresponding weights. Performance.

[0066] 3) According to Feedback is used to execute selection, crossover, and mutation operators to generate the next generation of weighted populations. .

[0067] 4) Determine whether the termination condition has been met (reaching the maximum number of generations or the fitness tends to stabilize).

[0068] 5) Extract the weight combination with the highest fitness in the evolutionary history. This is then applied to the final training of the fusion model.

[0069] Traditional fixed weights (such as 1:1:1:1) cannot adapt to different scenarios. Through evolutionary algorithms, the model can discover appropriate detection losses in certain scenarios. The weights need to be significantly amplified to improve performance. This avoids the tedious process of researchers adjusting weights through trial and error, greatly improving the efficiency of algorithm deployment. The evolutionary algorithm can automatically find the "Pareto Front," that is, to obtain the highest machine detection performance while ensuring the visual naturalness of the image.

[0070] Table 1 shows the comparison of evaluation metrics for the fusion results. Table 2 shows the object detection results. Nine models were compared in the experiment: DDFM, YDTR, ReCoNet, LRRNet, SwinFusion, CDDFuse, EMMA, Text-IF, and SegMiF. Six metrics were used.

[0071] Table 1 Comparison of Evaluation Indicators for Fusion Results ; Table 2 Target Detection Results Data Table ; Therefore, this embodiment has the following advantages compared to the prior art: 1) The introduction of the Mamba model significantly improves the ability to model large targets and overcomes the feature loss caused by the limited receptive field of traditional convolutional networks.

[0072] 2) The use of "explicit injection" instead of "gradient backpropagation" solves the problem of interference with fusion quality in the absence of significant target samples, resulting in better visual effects of the generated images.

[0073] 3) Wavelet transform is used to differentiate the target saliency and background texture in the frequency domain, effectively preserving infrared thermal energy distribution and visible light details.

[0074] 4) The introduction of evolutionary strategies eliminates the tedious manual parameter tuning process, enabling the model to spontaneously evolve optimal parameters for different detection tasks.

[0075] In summary, as Figure 3 As shown, this embodiment aims to improve the detection accuracy of fused images in downstream target detection tasks by guiding the cross-modal feature fusion process through an explicit task awareness mechanism. First, multimodal feature extraction is performed: a hybrid encoder is used to extract features from normalized infrared and visible light images. A state-space model is introduced to capture the long-range spatial dependencies of the images, and a convolutional neural network is combined to extract local detail features, achieving a joint representation of global semantics and local structure. Next, multi-scale decomposition is performed: discrete wavelet transform is used to decompose the extracted features into low-frequency components containing the main energy distribution of the image and high-frequency components containing texture and edge information, achieving decoupled representation of spatial and frequency domain features. Further, task-aware enhancement is performed: a pre-trained object detection network is used to extract deep semantic features related to the detection task as explicit task guidance signals. These semantic features are then injected into each level of the wavelet decomposition components through a cross-modal attention mechanism, achieving directional feature enhancement for object perception. Finally, feature fusion and reconstruction are performed: the high and low frequency components enhanced by task perception are differentially and adaptively fused, and the final fused image is generated through inverse wavelet transform and a residual reconstruction network. Joint optimization is also included: a joint loss function composed of image fusion loss and object detection loss is constructed, and an evolutionary strategy is used to globally optimize hyperparameters such as loss weights to achieve an adaptive balance between fusion quality and object detection performance. This embodiment effectively improves the perception capability of the fused image of the target region by explicitly introducing semantic prior information of the target detection task, and significantly improves the performance of the downstream target detection task while ensuring the visual quality of the fused image.

[0076] Reference Figure 2 An image fusion system based on cross-modal attention explicit task guidance includes: The first module 201 is used to normalize the infrared image and the visible light image and input them into the state space model and the convolutional neural network model respectively for feature extraction to obtain the infrared image features and the visible light image features. The second module 202 is used to perform feature decomposition processing on infrared image features and visible light image features using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. The third module 203 is used to generate explicit task guidance signals and inject the explicit task guidance signals into the low-frequency components containing the main contours and the high-frequency components containing texture edges through a cross-modal attention mechanism to perform feature enhancement processing for target perception, thereby obtaining enhanced high and low frequency components. The fourth module 204 is used to classify and adaptively fuse the enhanced high and low frequency components, and reconstruct the fusion features through wavelet inverse transform to obtain the target perception fusion image.

[0077] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0078] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image fusion method based on cross-modal attention explicit task guidance, characterized in that, Includes the following steps: Infrared and visible light images are normalized and then input into a state-space model and a convolutional neural network model, respectively, for feature extraction to obtain infrared image features and visible light image features. The infrared and visible light image features are decomposed using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. An explicit task guidance signal is generated, and the explicit task guidance signal is injected into the low-frequency component containing the main contour and the high-frequency component containing the texture edge through a cross-modal attention mechanism to perform feature enhancement processing for target perception, resulting in enhanced high and low frequency components. The enhanced high and low frequency components are classified and adaptively fused, and the fused features are reconstructed through wavelet inverse transform to obtain the target perception fused image.

2. The image fusion method based on cross-modal attention explicit task guidance according to claim 1, characterized in that, The step of normalizing the infrared and visible light images and inputting them into the state-space model and convolutional neural network model respectively for feature extraction to obtain infrared image features and visible light image features specifically includes: Two-dimensional infrared images and two-dimensional visible light images are acquired and converted to obtain one-dimensional vector sequences of infrared images and visible light images. The infrared image feature map and the visible light feature map are obtained by sequentially performing image mapping and image flattening processing on the one-dimensional vector sequence of the infrared image and the one-dimensional vector sequence of the visible light image through the overlapping block method. Based on the state-space model, the SS2D module is introduced to perform four-way feature scanning and extraction on the infrared feature map and the visible light feature map to obtain the infrared global feature and the visible light global feature. By using a convolutional neural network model to perform local feature constraint processing on global infrared and global visible light features, infrared image features and visible light image features are obtained.

3. The image fusion method based on cross-modal attention explicit task guidance according to claim 2, characterized in that, The step of performing feature decomposition processing on infrared and visible light image features using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges specifically includes: Define discrete wavelet basis functions and calibration functions and perform inner product calculations to obtain detail coefficients and approximation coefficients, and construct the decomposition expression of the wavelet basis signal at a preset scale; By introducing low-frequency adaptive analysis vectors and high-frequency adaptive analysis vectors, and combining them with the decomposition expression of wavelet basis signals at a preset scale, four sets of convolution operators are constructed. The four sets of convolution operators include low-frequency, horizontal high-frequency, vertical high-frequency, and diagonal high-frequency convolution operators. Four sets of convolution operators are used to convolve and downsample the infrared image features and visible light image features in the row and column directions, respectively, to obtain the first-level low-frequency approximation component, the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component and the first-level high-frequency diagonal detail component. The DWT filter bank is used to perform secondary decomposition on the first-level low-frequency approximation component, the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, and the first-level high-frequency diagonal detail component to obtain the second-level low-frequency approximation component, the second-level high-frequency horizontal detail component, the second-level high-frequency vertical detail component, and the second-level high-frequency diagonal detail component. The second-order low-frequency approximation component is injected into the global semantics of the target detection network to construct a low-frequency component containing the main contour; By employing a spatial activity adaptive enhancement strategy, local variance calculations are performed on the first-level high-frequency horizontal detail component, the first-level high-frequency vertical detail component, the first-level high-frequency diagonal detail component, the second-level high-frequency horizontal detail component, the second-level high-frequency vertical detail component, and the second-level high-frequency diagonal detail component to obtain high-frequency components that include texture edges.

4. The image fusion method based on cross-modal attention explicit task guidance according to claim 3, characterized in that, The step of generating an explicit task guidance signal and injecting it into the low-frequency component containing the main contour and the high-frequency component containing texture edges through a cross-modal attention mechanism for target perception feature enhancement processing to obtain enhanced high and low frequency components specifically includes: The detected image is input into a pre-trained target detection network, the intermediate layer feature maps of its deep backbone network are extracted, and an explicit task guidance signal is generated after dimension alignment. Construct parallel cross-dimensional attention modules, using low-frequency components containing the main contour and high-frequency components containing texture edges as query terms, and explicit task guidance signals as key and value terms. By learning the weight matrix, linear projections are performed on the query item, key item, and value item respectively, and the correlation between each pixel of the wavelet component and the inspection prompt signal is calculated, target perception attention is generated. By using a residual injection structure, target perception attention is injected into the low-frequency component containing the main contour and the high-frequency component containing the texture edge for target perception feature enhancement processing, resulting in enhanced high and low frequency components.

5. The image fusion method based on cross-modal attention explicit task guidance according to claim 4, characterized in that, The step of classifying and adaptively fusing the enhanced high and low frequency components, and reconstructing the fused features through wavelet inverse transform to obtain the target-aware fused image, specifically includes: An adaptive strategy based on energy weights is used to fuse the enhanced low-frequency components to obtain the fused low-frequency components. Based on the spatial activity selection strategy, the enhanced high-frequency components are fused to obtain the fused high-frequency components. The low-frequency component and the high-frequency component after fusion are reconstructed by multi-scale spatial reconstruction using the inverse discrete wavelet transform method to obtain the reconstructed feature components. The reconstructed feature components are input into a decoding network containing residual connections, and mean information from infrared and visible light images is introduced through long-range skip connections to perform global contrast restoration and noise suppression, resulting in a target perception fusion image.

6. The image fusion method based on cross-modal attention explicit task guidance according to claim 5, characterized in that, The specific expression for the energy-weight-based adaptive strategy is as follows: In the above formula, This represents the low-frequency component after fusion. Represents the low-frequency component of infrared radiation. This represents the low-frequency component of visible light. , This represents the weighting coefficient.

7. The image fusion method based on cross-modal attention explicit task guidance according to claim 6, characterized in that, The specific expression for the spatial activity-based selection strategy is as follows: In the above formula, This represents the high-frequency components after fusion. Represents the low-frequency component of infrared radiation. This represents the low-frequency component of visible light. This indicates taking the maximum value element by element.

8. The image fusion method based on cross-modal attention explicit task guidance according to claim 7, characterized in that, The specific expression of the multi-objective optimization function for the target-aware fusion image is as follows: In the above formula, Represents a multi-objective optimization function. Indicates strength loss. Represents gradient loss, Represents structural similarity loss. Indicates detection loss, , , , This represents the weighting parameter.

9. An image fusion system based on explicit task guidance of cross-modal attention, characterized in that, Includes the following modules: The first module is used to normalize infrared and visible light images and input them into the state space model and convolutional neural network model respectively for feature extraction to obtain infrared image features and visible light image features. The second module is used to perform feature decomposition processing on infrared image features and visible light image features using the discrete wavelet transform method to obtain low-frequency components containing the main contours and high-frequency components containing texture edges. The third module is used to generate explicit task guidance signals and inject the explicit task guidance signals into the low-frequency components containing the main contours and the high-frequency components containing texture edges through a cross-modal attention mechanism to perform feature enhancement processing for target perception, resulting in enhanced high and low frequency components. The fourth module is used to classify and adaptively fuse the enhanced high and low frequency components, and reconstruct the fusion features through wavelet inverse transform to obtain the target-aware fusion image.

Images