Metalearning multi-modal image fusion method and system facing downstream perception task

Abstract

The invention provides a meta-learning multi-modal image fusion method and system for a downstream perception task, and belongs to the field of multi-source perception and image processing. The problems that existing deep fusion depends on image-level loss of fixed design, coupling with downstream task performance is neglected, and it is difficult to highlight a target area and suppress a background at the same time are solved. The method comprises a fusion network, a loss generation network and a task network ternary closed loop, wherein a fusion branch extracts a low-frequency structure through discrete wavelets and multi-scale global memory, and enhances high-frequency details through local window multi-head self-attention guided by dynamic Gabor; and finally, performing judgment-level refining and outputting a fused image by combining local attention through base fusion of related gating and pixel-by-pixel amplitude self-adaptive detail fusion. According to the method, the performance of downstream tasks such as segmentation and detection is remarkably enhanced while the structural consistency and detail definition of the fused image are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-source perception and image processing, in particular to a meta-learning multi-modal image fusion method and system for downstream perception tasks. BACKGROUND

[0002] Infrared images have the advantages of weak light robustness, strong night vision capability, and insensitivity to light changes because they are formed by receiving the infrared energy radiated by objects. They can maintain stable imaging in complex scenes such as night, backlight, rain and fog. The strong thermal target features make them irreplaceable in the fields of security monitoring, autonomous driving, border patrol, and unmanned reconnaissance. In contrast, visible light images are based on natural light reflection imaging and have high spatial resolution, rich texture details, and complete structural information. They have obvious advantages in edge, texture, and detail reconstruction. Due to the complementarity of the two modalities in imaging mechanism, noise distribution, and salient features, the fusion of infrared and visible light can achieve robust and high-quality scene perception, so it has become an important research direction in multi-modal perception in recent years.

[0003] Existing deep learning-driven image fusion methods mainly rely on artificially designed image-level loss functions, such as gradient loss, structural similarity loss, and reconstruction loss, to constrain the fusion results. However, these fixed losses generally lack direct correlation with downstream tasks, leading to a disconnect between optimization objectives and actual application needs. In complex backgrounds or multi-target scenes, such methods often struggle to simultaneously highlight target regions and suppress background regions, thereby affecting detection and recognition performance. A class of feature-driven methods attempts to incorporate detection features, heat maps, or semantic guidance information into the fusion process. However, the weak coupling between these guiding features and the fusion network can amplify false saliencies and even affect fusion stability. Another class of loss-driven methods directly backpropagate task network losses to the fusion network. However, due to significant differences in data distribution, numerical scale, and gradient variation between infrared and visible light, this optimization process is often unstable and prone to oscillation. Moreover, it may sacrifice visualization quality to meet task loss, leading to degradation in texture and detail on the fused image.

[0004] Current techniques still face contradictions between visual quality, task performance, and training stability, lacking a unified fusion framework that can accommodate multi-modal characteristics while adapting to downstream tasks such as detection and segmentation. Therefore, there is an urgent need to design a task-friendly fusion paradigm with stable optimization capabilities that can significantly improve detection accuracy, segmentation accuracy, and overall scene understanding while ensuring image subjective quality, thereby meeting the practical needs of intelligent security, autonomous driving, and unmanned systems. SUMMARY

[0005] The technical problem to be solved by this invention is: To address the issue that existing deep fusion methods rely on fixed-design image-level loss, neglect the coupling with downstream task performance, and struggle to simultaneously highlight the target region and suppress the background.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: This invention provides a meta-learning multimodal image fusion method for downstream perception tasks, comprising the following steps: S100, Data Acquisition and Preprocessing: Acquire registered infrared and visible light images as input, and normalize and align the inputs. S200: Wavelet decomposition is performed on the two modes respectively, and multi-scale global memory is constructed to extract cross-scale robust global low-base features to obtain global structural information of the image, providing a foundation for subsequent high-frequency detail modeling. S300: Perform learnable dynamic Gabor decomposition and local window multi-head self-attention on the two-modal features to obtain direction-selective high-frequency detail enhancement features. S400 performs basal fusion based on Pearson correlation gating, performs detail fusion based on pixel-wise amplitude scaling, and outputs the fused image through decision-level refinement using convolution and local attention. S500. Construct a loss generation module LGM, which includes a small number of scalar parameters and iteration steps. Construct pixel error using two modes and their mean map, and perform closed-loop initialization. S600, Joint Loss Construction and Weighting, including calculating fusion loss and task loss, and calculating total loss; S700 employs a meta-learning training strategy, utilizing a dual-loop meta-learning approach within each training round. The inner loop freezes the loss generation module (LGM) and updates the fusion network and task network using pixel / global weights. The outer loop freezes the fusion and task branches, optimizing the LGM with the task loss as the meta-objective and performing gradient pruning. Subsequently, in round-level re-estimation, the algorithm is based on the loss reduction rate of adjacent rounds. The weights are updated and used in the next round of training to mitigate the oscillations and biases caused by scale mismatch.

[0007] Further, in step S100, the following are included: S110. Acquire time-synchronized and spatially registered infrared images. With visible light images It performs distortion correction, coarse registration, and fine registration to unify the two modes to the same size. ; S120. Perform intensity normalization and noise suppression on the two modes respectively to obtain the normalized input. ; S130, Normalize the input Input the two branches of the fusion network.

[0008] Further, in step S200, the following are included: S210. Perform two-dimensional separable Haar wavelet discretization on the two modal inputs respectively to obtain the approximate low-frequency subband. With high-frequency subband :

[0009]

[0010]

[0011] in, These are low-pass and high-pass filters, respectively; symbol , This indicates the filter's flipped or transposed form in another direction; This represents the 2x downsampling operator; DWT(x) represents the two-dimensional discrete wavelet transform of the input image x. S220, in the low-frequency subband A multi-scale global memory is constructed, and the global context is extracted in parallel at five scales: each scale is processed by 1×1 convolution, BN normalization, ReLU activation function, 1×1 convolution, BN normalization, and ReLU activation function to obtain the global memory features. A multi-scale feature pool is formed by splicing channels. :

[0012] Among them, G J This represents the global structural feature corresponding to the J-th scale, where the scale number J∈{1,2,3,4,5}.

[0013] S230, for cross-scale feature pools Two-stage 1×1 convolution compression is performed. The mapping head integrates local context and linear projection through a 3×3 convolution and a 1×1 convolution, further mapping the compressed features into a single-channel basic feature map. The basis diagrams for the two modes are denoted as follows: .

[0014] Further, in step S300, the following are included: S310. Perform dynamic Gabor decomposition on the input, for each direction. Learning a set of parameters that are physically interpretable ,in, For direction number; scale parameters Ensure that it is at the lower bound. Limit to ; where Softplus indicates applying the Softplus activation function to the parameters; p min This indicates the positive lower bound set for the parameter; tanh indicates applying the tanh activation function to the parameter.

[0015] S320, Construct real / virtual Gabor kernels in each direction. , Then, convolution is used to obtain real / virtual Gabor kernel responses. ; Softmax weights in the meridional direction By fusion, the Gabor basis map is obtained. Detailed images of Gabor ; S330, with detailed Gabor images Generate as a query Using Gabor base map Generate as key / value After mirror filling, the features are divided into window side lengths. Non-overlapping local windows, performing multi-head self-attention within the windows:

[0016] Where ω represents the local window number currently being processed; Q ω ,K ω V ω These represent the Query, Key, and Value features extracted within window ω, respectively; d represents the channel dimension of each attention head, i.e., the vector length of Query / Key. After residual MLP and Projection yields a detail-enhanced image Let the details of the two modes be denoted as follows: .

[0017] Further, in step S400, the following is included: S410, based on Pearson correlation gating, performs basis fusion, including computation. Pearson correlation coefficient The gating coefficient is obtained by passing through two layers of perceptrons and the sigmoid function. Output:

[0018] Among them, B f The base features after fusion; S420: Detail fusion is performed based on pixel-by-pixel amplitude scaling, including generating pixel-level scaling based on the relative magnitudes of the two-modal detail amplitudes.

[0019]

[0020] Where α represents the pixel-level fusion weighting coefficient for infrared and visible light detail amplitudes; ε represents a very small constant used to avoid division by zero; D f This represents the feature map of the final detail fusion result obtained according to the weight α; S430, Convolutional Blending and Decision-Level Refinement: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] go through Convolution, BN normalization, and ReLU activation function, cascaded and then... Obtain the initial fusion diagram ;right respectively Convolution with input local attention operator The channel GAP global average pooling, 3×3 convolution, BN normalization, ReLU activation function, 1×1 convolution and Sigmoid function are used to obtain weighted features, which are normalized to [0,1] to output the fusion result Y.

[0021] Further, in step S500, the following is included: S510. Construct the two-mode diagram and its mean plot. Pixel error for:

[0022] Initial weights are obtained by performing closed-form initialization. :

[0023] S520, Introducing Isotropic Properties The regular discrete gradient approximation forms the energy gradient. :

[0024] With energy gradient Perform explicit descent and update via interval clipping:

[0025] Here, `clip!` represents a clipping operation, which restricts the input value to a specific range. Within; any step size η∈(0,1); Let be the weight for the t-th iteration; Iteration The next output is pixel-level dual weights. And obtained through channel normalization .

[0026] Further, in step S600, the following are included: S610. Calculate the fusion loss by aligning the weighted intensity MSE with the maximum gradient:

[0027] in, For Sobel horizontal and vertical operators; ∥1 denotes the L1 norm; S620. Calculate the task loss through semantic segmentation and object detection: Semantic segmentation, Where N is the number of pixels; For the first i The real label of each pixel; Let $\frac{i}{i}$ be the class probability predicted by the model for the $i$-th pixel. Target detection, ;in, Used to control bounding box regression loss Weighting coefficients; Weights used to control the classification loss; Weighting coefficients used to control the loss of distributed regression; S630, Calculate the total loss: ,in Based on the specific downstream task, it is divided into cross-entropy.

[0028] Further, in step S700, the following is included: S710, inner loop, freeze loss generation module LGM, calculates the total loss using pixel-level weights and the current global weights. The fusion network and the task network are respectively categorized by learning rate. Gradient descent is used to jointly update the fusion network and the downstream task network. S720, outer ring, frozen fusion and mission branches, with mission loss Optimize the LGM parameters of the loss generation module for the meta-objective; S730, Round-by-Round Reassessment: Round-by-round reassessment uses temperature parameters (T>0) and is based on the loss reduction rate of adjacent rounds. The new round of global weights is calculated using Softmax. ;in, and These represent the decrease rates of fusion loss and task loss in adjacent rounds, respectively. T For temperature parameters; This represents the fusion loss in round t; This represents the task loss in round t.

[0029] A meta-learning multimodal image fusion system for downstream perception tasks is provided. The system has program modules corresponding to the steps described above, and executes the steps in the meta-learning multimodal image fusion method for downstream perception tasks described above when running.

[0030] A computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement steps of a meta-learning multimodal image fusion method for downstream perception tasks.

[0031] Compared with the prior art, the beneficial effects of the present invention are: This invention seamlessly injects task semantics into the fusion process through a three-stage progressive fusion of "structure-detail-decision" and dual adaptive supervision of "pixel-global". On the one hand, it constructs an interference-resistant global structural base with DWT+MSGM and achieves directional selective detail enhancement with dynamic Gabor+local window attention, significantly improving the structural consistency and detail clarity of the fused graph. On the other hand, it generates pixel-level and global weights with interpretable energy descent using LGM, and combines inner and outer double loops + round re-estimation for stable training, significantly improving the discriminative power and robustness of downstream tasks such as segmentation and detection, while maintaining lightweight efficiency and good deployability. Attached Figure Description

[0032] Figure 1 This is a flowchart of a meta-learning multimodal image fusion method for downstream perception tasks in an embodiment of the present invention; Figure 2 This is a schematic diagram of the enhanced substrate extraction (EBE) structure in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the dynamically Gabor-guided local window multi-head self-attention (GG-MHA) in an embodiment of the present invention. Detailed Implementation

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0034] Specific Implementation Plan 1: Combining Figures 1 to 3 As shown, this invention provides a meta-learning multimodal image fusion method for downstream perception tasks, comprising the following steps: S100, Data Acquisition and Preprocessing: Acquire registered infrared and visible light images as input, and normalize and align the inputs; including... S110. Acquire time-synchronized and spatially registered infrared images. With visible light images It performs distortion correction, coarse registration, and fine registration (such as feature point-based + subpixel optical flow fine-tuning) to unify the two modalities to the same size. ; S120. Perform intensity normalization and noise suppression (such as guided filtering / bilateral filtering) on ​​the two modes respectively to obtain the normalized input. ; S130, Normalize the input The two branches of the input fusion network are used to prepare for subsequent low-frequency structure and high-frequency detail modeling; S200, Perform global low-frequency feature extraction (EBE: DWT+MSGM, such as...) Figure 2 This is used to obtain global structural information of the image; wavelet decomposition is performed on both modalities and a multi-scale global memory is constructed to extract cross-scale robust low-frequency basis features; including, S210. Perform two-dimensional separable Haar wavelet discretization on the two modal inputs respectively to obtain the approximate low-frequency subband. With high-frequency subband :

[0035]

[0036]

[0037] in These are low-pass and high-pass filters, respectively; symbol , This indicates the filter's flipped or transposed form in another direction; This represents the 2x downsampling operator; DWT(x) represents the two-dimensional discrete wavelet transform of the input image x. S220, in the low-frequency subband Multi-scale Global Memory (MSGM) is constructed, and global context is extracted in parallel at 5 scales: each scale is processed by "1×1 convolution - BN normalization - ReLU activation function - 1×1 convolution - BN normalization - ReLU activation function" to obtain global memory features. A multi-scale feature pool is formed by splicing channels. :

[0038] Among them, G J Let J be the global structural feature corresponding to the J-th scale, where the scale number J∈{1,2,3,4,5}; S230, for cross-scale feature pools Two-stage 1×1 convolution compression is performed. The mapping head integrates local context and linear projection through a 3×3 convolution and a 1×1 convolution, further mapping the compressed features into a single-channel basic feature map. Let the basis diagrams of the two modes be denoted as follows: ; S300, to fully capture directional details, textures, and high-frequency structures in an image, while avoiding the high computational cost and noise amplification caused by global self-attention modeling across the entire image, performs direction-sensitive detail extraction (GGMHA, such as...). Figure 3 First, the two-modal input is decomposed into a basal layer and a detail layer using learnable Gabor decomposition. Then, the detail layer generates the query Q, and the basal layer generates the key-value pairs K / V. Multi-head attention is performed within a local window, thereby achieving adaptive enhancement of details and effective control of computational complexity. S310. Perform Dynamic Gabor Decomposition (LGD) on the input for each direction. Learning a set of parameters that are physically interpretable ,in, For direction number; To ensure learnability and numerical stability, the scale parameter is... by Ensure that it is at the lower bound. by Limit to ; where Softplus indicates applying the Softplus activation function to the parameters; p min This indicates the positive lower bound set for the parameter to ensure that the final result is always greater than 0, avoiding numerical instability or degradation; tanh indicates that the tanh activation function is applied to the parameter; S320, Construct each direction Real / virtual Gabor cores , Then, convolution is used to obtain real / virtual Gabor kernel responses. To integrate responses from multiple directions, a learnable directional attention mechanism is introduced, using directional softmax weights. By fusion, the Gabor basis map is obtained. Detailed images of Gabor ; S330, with detailed Gabor images Generate as a query Using Gabor base map Generate as key / value After mirror filling, the features are divided into window side lengths. Non-overlapping local windows, performing multi-head self-attention within the windows:

[0039] Where ω represents the local window number currently being processed, used to distinguish different local attention regions; Q ω ,K ω V ω These represent the Query, Key, and Value features extracted within window ω, respectively; d represents the channel dimension of each attention head, i.e., the vector length of Query / Key, which is a regular dimensional constant. After residual MLP and Projection yields a detail-enhanced image Let the details of the two modes be denoted as follows: ; The local window multi-head self-attention performs mirror filling on the features before calculation, so that the height and width of the features are integer multiples of the window side length, and calculates attention independently in each non-overlapping window. S400 aims to effectively suppress statistical inconsistencies in cross-modal basis component fusion and noise amplification in detail component fusion. It also implements an end-to-end, learnable, and lightweight final reconstruction head for final fusion (Base–Detail Fusion Layer, BDFL: correlation gating + pixel-wise magnitude adaptation + decision-level refinement). This includes base fusion based on Pearson correlation gating, detail fusion based on pixel-wise magnitude scaling, and decision-level refinement using convolution and local attention to output the fused image. Specifically, including, S410. Considering the potential statistical inconsistencies between infrared and visible light at low frequencies, the fusion weights (correlation gating) are determined based on their Pearson correlation coefficient: Calculation Pearson correlation coefficient The gating coefficient is obtained by passing through two layers of perceptrons and the sigmoid function. Output:

[0040] Among them, B f The base features after fusion; The gating coefficients of the relevant gating are output by two layers of perceptrons and limited to (0,1) by Sigmoid, which are used to suppress artifacts caused by cross-modal low-frequency statistical inconsistencies; S420, infrared, and visible light images typically exhibit strong local contrast differences. Details with strong responses dominate at their corresponding locations. Fixed weighting or simple stitching can indirectly lead to detail loss or noise introduction. To obtain fine and natural blended details, the fusion ratio is determined by the relative magnitudes of the two modal detail amplitudes, thus achieving a parameter-free, pixel-by-pixel fusion strategy. Specifically, detail fusion is completed based on pixel-by-pixel amplitude ratios: pixel-level ratios are generated by the relative magnitudes of the two modal detail amplitudes.

[0041]

[0042] Where α represents the pixel-level fusion weighting coefficient for infrared and visible light detail amplitudes, with a value ranging from 0 to 1; ε represents a very small constant used to avoid division by zero; D f This represents the feature map of the final detail fusion result obtained according to the weight α; S430, Convolutional Blending and Decision-Level Refinement: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] go through Convolution, BN normalization, and ReLU activation function, cascaded and then... Obtain the initial fusion diagram ;right respectively Convolution with input local attention operator The channel GAP global average pooling, 3×3 convolution, BN normalization, ReLU activation function, 1×1 convolution and Sigmoid function are used to obtain weighted features, which are then normalized to ([0,1]) to output the fusion result. ; S500 constructs a loss generation module (LGM) with pixel-level and globally adaptive weights to ensure that the fused image retains the salient features of both infrared and visible light images. The loss generation module (LGM) includes a small number of scalar parameters. The iteration step number K, where β is the TV regularization coefficient and η is the iteration step size. A scalable energy adjustment parameter is used to control the shape of weight updates; pixel error is constructed using two modes and their mean map. And perform closed initialization; including, S510. Construct the two-mode diagram and its mean plot. Pixel error for:

[0043] Initial weights are obtained by performing closed-form initialization. : ; S520, Introducing Isotropic Properties The regular discrete gradient approximation forms the energy gradient. :

[0044] With energy gradient Perform explicit descent and update via interval clipping:

[0045] Here, `clip!` represents a clipping operation, which restricts the input value to a specified upper and lower bound range. Within (0,1); any step size η∈(0,1); Let be the weight for the t-th iteration; Represents the gradient; Iteration The next output is pixel-level dual weights. And obtained through channel normalization ; S600, joint loss construction and weighting; including, S610. Calculate the fusion loss using the weighted intensity MSE and the maximum gradient consistency (Sobel):

[0046] in, For Sobel horizontal and vertical operators; ∥1 denotes the L1 norm; S620. Calculate the task loss through semantic segmentation and object detection: For segmentation tasks, to effectively improve the accuracy of pixel-level classification, this invention uses cross-entropy loss, widely used in semantic segmentation tasks, as a supervision signal. This effectively optimizes the model's classification boundary, thereby improving the overall performance of semantic segmentation. The loss function for semantic segmentation is expressed as:

[0047] Where N is the number of pixels; For the first i The real label of each pixel; Let $\frac{i}{i}$ be the class probability predicted by the model for the $i$-th pixel. Detection Task. In the object detection branch, this invention adopts a YOLO-style multi-component loss structure, unifying bounding box regression and class classification into a single model. The loss function for object detection is expressed as:

[0048] in, Used to control bounding box regression loss Weighting coefficients; Used to control classification loss The weights; Used to control distributed regression loss Weighting coefficients; S630, Calculate the total loss: ,in Based on the specific downstream task, it is divided into cross-entropy; S700, designing meta-learning training strategies (inner and outer double loops + round-by-round re-estimation, such as...) Figure 1 The overall training process is a two-layer loop, employing inner and outer loop meta-learning within each training round: the inner loop freeze loss generation module (LGM) utilizes pixel / global weighted averages. Update the fusion network and the task network; freeze the fusion and task branches in the outer loop, optimize the LGM with the task loss as the meta-objective, and perform gradient pruning; subsequently, in the round-level re-evaluation, based on the loss reduction rate of adjacent rounds... pass renew This is used in the next round of training to mitigate oscillations and biases caused by scale mismatch; including S710, inner loop (update fusion / task branch): freeze the loss generation module LGM, using pixel-level weights and the current global weights. Calculate total loss The fusion network and the task network are respectively categorized by learning rate. Gradient descent is used to jointly update the fusion network and the downstream task network. S720, Outer Ring (Update Loss Generation Module LGM): Freeze fusion and task branches, based on task loss. Optimize the LGM parameters of the loss generation module for the meta-objective (using gradient pruning to improve stability); S730, Round-by-Round Reassessment: Round-by-round reassessment uses the temperature parameter (T>0) and is based on the loss reduction rate of adjacent rounds. The new round of global weights is calculated using Softmax. ;in, and These represent the rate of decrease of fusion loss and task loss in adjacent rounds, respectively; T=0.5 is the temperature parameter. Let t represent the fusion loss in round t. denoted as the task loss in round t, such as segmentation cross-entropy or detection IoU+BCE+DFL, is used to measure the performance of downstream tasks. S800, Inference Phase and Deployment, are used to demonstrate how the fusion model is used in the actual inference or deployment phase after training is completed, including the process of model loading, input preprocessing, performing forward inference and outputting fused images. S810, During inference, only S100~S400 are executed (can be selected in a fixed order). (Obtain the fastest path with LGM updates disabled) Output fusion graph It should be noted that the operations in S500 to S700 (including LGM update, meta-learning inner and outer loops, dynamic weight re-evaluation, etc.) are only used to provide auxiliary supervision during the training phase. The purpose is to allow the model to continuously adjust the weights and optimize the loss structure during training, thereby learning a better fusion strategy. In the actual inference phase, the model parameters have been trained and these update operations are no longer needed. Therefore, only the forward inference process in S100 to S400 needs to be executed to generate the fused image. S820, in the "fusion + task" coordinated deployment scenario, fusion graph The results can be directly fed into the downstream segmentation / detection network to obtain the task results; alternatively, a fused graph can be output. For human sensing applications; S830, typical configuration: , , , , , Without changing the essence of the method, the above parameters can be adjusted within a given range to adapt to different data distributions and computing power budgets.

[0049] Specific Implementation Method Two: The present invention provides a meta-learning multimodal image fusion system for downstream perception tasks. This system has program modules corresponding to the above steps, and executes the steps in the above-described meta-learning multimodal image fusion method for downstream perception tasks during runtime.

[0050] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0051] Specific Implementation Method 3: The present invention provides a computer-readable storage medium storing a computer program configured to implement, when called by a processor, the steps of a meta-learning multimodal image fusion method for downstream perception tasks.

[0052] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0053] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A meta-learning multimodal image fusion method for downstream perception tasks, characterized in that, Includes the following steps: S100, Data Acquisition and Preprocessing: Acquire registered infrared and visible light images as input, and normalize and align the inputs. S200: Wavelet decomposition is performed on the two modes respectively, and multi-scale global memory is constructed to extract cross-scale robust global low-base features to obtain global structural information of the image, providing a foundation for subsequent high-frequency detail modeling. S300: Perform learnable dynamic Gabor decomposition and local window multi-head self-attention on the two-modal features to obtain direction-selective high-frequency detail enhancement features. S400 performs basal fusion based on Pearson correlation gating, performs detail fusion based on pixel-wise amplitude scaling, and outputs the fused image through decision-level refinement using convolution and local attention. S500. Construct a loss generation module LGM, which includes a small number of scalar parameters and iteration steps. Construct pixel error using two modes and their mean map, and perform closed-loop initialization. S600, Joint Loss Construction and Weighting, including calculating fusion loss and task loss, and calculating total loss; S700 designs a meta-learning training strategy, employing inner and outer double-loop meta-learning in each training round, including an inner loop frozen loss generation module (LGM), which uses pixel / global weights to weight and update the fusion network and task network. The outer loop freezes the fusion and task branches, optimizes the loss generation module LGM with the task loss as the meta-objective, and performs gradient pruning; subsequently, in the round-level re-evaluation, the loss reduction rate of adjacent rounds is used as the basis for... The weights are updated and used in the next round of training to mitigate the oscillations and biases caused by scale mismatch.

2. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 1, characterized in that: In step S100, the following are included: S110. Acquire time-synchronized and spatially registered infrared images. With visible light images It performs distortion correction, coarse registration, and fine registration to unify the two modes to the same size. ; S120. Perform intensity normalization and noise suppression on the two modes respectively to obtain the normalized input. ; S130, Normalize the input Input the two branches of the fusion network.

3. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 2, characterized in that: Step S200 includes, S210. Perform two-dimensional separable Haar wavelet discretization on the two modal inputs respectively to obtain the approximate low-frequency subband. With high-frequency subband : ； ； ； in These are low-pass and high-pass filters, respectively; symbol , This indicates the filter's flipped or transposed form in another direction; This indicates a 2x downsampling operator; DWT(x) represents a two-dimensional discrete wavelet transform of the input image x; S220, in the low-frequency subband A multi-scale global memory is constructed, and the global context is extracted in parallel at five scales: each scale is processed by 1×1 convolution, BN normalization, ReLU activation function, 1×1 convolution, BN normalization, and ReLU activation function to obtain the global memory features. A multi-scale feature pool is formed by splicing channels. : ； Among them, G J Let J be the global structural feature corresponding to the J-th scale, where the scale number J∈{1,2,3,4,5}; S230, for cross-scale feature pools Two-stage 1×1 convolution compression is performed. The mapping head integrates local context and linear projection through a 3×3 convolution and a 1×1 convolution, further mapping the compressed features into a single-channel basic feature map. Let the basis diagrams of the two modes be denoted as follows: .

4. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 3, characterized in that: Step S300 includes, S310. Perform dynamic Gabor decomposition on the input, for each direction. Learning a set of parameters that are physically interpretable ,in, For direction number; scale parameters Ensure that it is at the lower bound. Limit to Where Softplus indicates that the Softplus activation function is applied to the parameters; p min This indicates the positive lower bound set for the parameter; tanh indicates applying the tanh activation function to the parameter. S320, Construct real / virtual Gabor kernels in each direction. , Then, convolution is used to obtain real / virtual Gabor kernel responses. ; Softmax weights in the meridional direction By fusion, the Gabor basis map is obtained. Detailed images of Gabor ; S330, with detailed Gabor images Generate as a query Using Gabor base map Generate as key / value After mirror filling, the features are divided into window side lengths. Non-overlapping local windows, performing multi-head self-attention within the windows: ； Where ω represents the local window number currently being processed; Q ω ,K ω V ω represents the Query, Key, and Value features extracted within window ω, respectively; d represents the channel dimension of each attention head, i.e., the vector length of Query / Key; after residual MLP and Projection yields a detail-enhanced image Let the details of the two modes be denoted as follows: .

5. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 4, characterized in that: In step S400, the following are included: S410, based on Pearson correlation gating, performs basis fusion, including computation. Pearson correlation coefficient The gating coefficient is obtained by passing through two layers of perceptrons and the sigmoid function. Output: ； Among them, B f The base features after fusion; S420: Detail fusion is performed based on pixel-by-pixel amplitude scaling, including generating pixel-level scaling based on the relative magnitudes of the two-modal detail amplitudes. ； ； Where α represents the pixel-level fusion weighting coefficient for infrared and visible light detail amplitudes; ε represents a very small constant used to avoid division by zero; D f This represents the feature map of the final detail fusion result obtained according to the weight α; S430, Convolutional Blending and Decision-Level Refinement: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] go through Convolution, BN normalization, and ReLU activation function, cascaded and then... Obtain the initial fusion diagram ;right respectively Convolution with input local attention operator The channel GAP global average pooling, 3×3 convolution, BN normalization, ReLU activation function, 1×1 convolution and Sigmoid function are used to obtain weighted features, which are then normalized to ([0,1]) to output the fusion result. .

6. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 5, characterized in that: Step S500 includes, S510. Construct the two-mode diagram and its mean plot. Pixel error for: ； Initial weights are obtained by performing closed-form initialization. : ； S520, Introducing Isotropic Properties The regular discrete gradient approximation forms the energy gradient. : ； With energy gradient Perform explicit descent and update via interval clipping: ； Here, `clip!` represents a clipping operation, which restricts the input value to a specific range. Within; any step size η∈(0,1); Let be the weight for the t-th iteration; Represents the gradient; Iteration The next output is pixel-level dual weights. And obtained through channel normalization .

7. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 6, characterized in that: Step S600 includes, S610. Calculate the fusion loss by aligning the weighted intensity MSE with the maximum gradient: ； in, For Sobel horizontal and vertical operators; ∥1 denotes the L1 norm; S620. Calculate the task loss through semantic segmentation and object detection: Semantic segmentation, Where N is the number of pixels; For the first i The real label of each pixel; Let $\frac{i}{i}$ be the class probability predicted by the model for the $i$-th pixel. Target detection, ;in, Used to control bounding box regression loss Weighting coefficients; Used to control classification loss The weights; Used to control distributed regression loss Weighting coefficients; S630, Calculate the total loss: ,in Based on the specific downstream task, it is divided into cross-entropy.

8. The meta-learning multimodal image fusion method for downstream perception tasks according to claim 7, characterized in that: Step S700 includes, S710, inner loop, freeze loss generation module LGM, using pixel-level weights and current global weights. Calculate total loss The fusion network and the task network are respectively categorized by learning rate. Gradient descent is used to jointly update the fusion network and the downstream task network. S720, outer ring, frozen fusion and mission branches, with mission loss Optimize the LGM parameters of the loss generation module for the meta-objective; S730, Round-by-Round Reassessment: Round-by-round reassessment uses temperature parameters (T>0) and is based on the loss reduction rate of adjacent rounds. Softmax calculates the new round of global weights. ;in, and These represent the decrease rates of fusion loss and task loss in adjacent rounds, respectively. T For temperature parameters; This represents the fusion loss in round t; This represents the task loss in round t.

9. A meta-learning multimodal image fusion system for downstream perception tasks, characterized in that: The system has a program module corresponding to the steps of any one of the claims 1-8 above, and executes the steps in the above-described meta-learning multimodal image fusion method for downstream perception tasks when it is run.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the meta-learning multimodal image fusion method for downstream perception tasks as described in any one of claims 1-8.

Images