Meta-learning driven progressive image fusion method and system

Abstract

The application discloses a progressive image fusion method and system based on meta-learning driving and belongs to the technical field of image processing. The method comprises the following steps: acquiring an infrared image and a visible light image; inputting the infrared image and the visible light image into a progressive cascade network which comprises a first-level perception dual-domain fusion network and a second-level perception dual-domain fusion network connected in series; performing rough perception on the input image based on the first-level perception dual-domain fusion network to generate a preliminary fusion image and a rough semantic segmentation image, and generating a spatial attention mask based on the rough semantic segmentation image; performing fine inference on the input image based on the spatial attention mask and the second-level perception dual-domain fusion network to generate a fine fusion image and a fine semantic segmentation image; and aggregating the preliminary fusion image and the fine fusion image to obtain a final fusion image, and aggregating the rough semantic segmentation image and the fine semantic segmentation image to obtain a final semantic segmentation image. The application effectively solves the problem that texture details and noise suppression are difficult to balance.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a progressive image fusion method and system based on meta-learning. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] In the continuous development of intelligent sensing technology, multimodal image fusion technology has become an important research direction in the field of computer vision. Due to the differences in imaging mechanisms among different types of sensors, a single sensor often struggles to adapt to complex and changing environments. Infrared imaging sensors, by capturing thermal radiation information, possess superior robustness in low-light environments such as smoke and nighttime conditions; while visible light sensors can provide rich spatial resolution, texture details, and color information. Infrared-visible light image fusion technology aims to effectively integrate the complementary information of these two sensors to generate high-quality images that possess both high-contrast target features and clear texture details.

[0004] In recent years, deep learning-based methods, such as autoencoders (AE), generative adversarial networks (GAN), and Transformer technology, have become the mainstream approaches in the field of infrared and visible light image fusion. However, although numerous fusion algorithms have been proposed, most methods are designed based on a single-stage reconstruction paradigm, which often struggles to effectively handle complex texture details and multi-task requirements.

[0005] Existing single-stage fusion frameworks attempt to simultaneously preserve texture and inject semantic information within a unified feature space. However, the inventors discovered that texture details and thermal noise in infrared images often highly overlap in the frequency domain. In single-stage processing, enhancing texture details inevitably amplifies high-frequency noise; conversely, excessive noise suppression can lead to overly smoothed fine textures. This inherent conflict between optimization objectives makes it difficult for the model to achieve an ideal balance between contrast enhancement and noise suppression, resulting in blurred edges or artifacts in the fused image. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a progressive image fusion method and system based on meta-learning. By introducing a progressive cascading mechanism, a hierarchical perception dual-domain fusion module, and a task decoupling meta-optimization strategy, the quality of the fused image and the performance of downstream tasks are significantly improved.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides a progressive image fusion method based on meta-learning driven by the present invention, including: Acquire infrared and visible light images; Infrared and visible light images are input into a progressive cascaded network for image fusion and semantic segmentation. The progressive cascaded network comprises a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series. The first-level perceptual dual-domain fusion network performs coarse perception on the input image, generating a preliminary fused image and a coarse semantic segmentation map. A spatial attention mask is then generated based on the coarse semantic segmentation map. Finally, the spatial attention mask and the second-level perceptual dual-domain fusion network perform fine reasoning on the input image, generating a fine fused image and a fine semantic segmentation map. The initial fused image and the fine fused image are combined to obtain the final fused image, and the coarse semantic segmentation map and the fine semantic segmentation map are combined to obtain the final semantic segmentation map.

[0008] In at least one embodiment, generating a spatial attention mask based on a coarse semantic segmentation map specifically includes: performing max pooling on the coarse semantic segmentation map along the channel dimension to extract the class activation value with the strongest response at each pixel location; performing large kernel convolution on the pooled result to aggregate spatial context information; and mapping the output to the [0,1] interval through the Sigmoid activation function to generate a spatial attention mask.

[0009] In at least one embodiment, the first-level perceptual dual-domain fusion network and the second-level perceptual dual-domain fusion network employ different fusion strategies for shallow features and deep features; wherein, for shallow features, a frequency domain decoupling fusion strategy based on discrete wavelet transform is adopted; and for deep features, a semantically guided cross-attention fusion strategy is adopted.

[0010] In at least one embodiment, the frequency domain decoupling and fusion strategy based on discrete wavelet transform specifically includes: Two-dimensional discrete wavelet transform is used to decompose the input shallow features into low-frequency structural components and high-frequency texture components; For the low-frequency components of infrared and visible light images, an adaptive weight fusion is performed using a spatially gated fusion module; For the high-frequency components of infrared and visible light images, grouped depthwise convolution is used for feature enhancement respectively; The fused low-frequency component is reconstructed by inverse wavelet transform with the enhanced visible light high-frequency component and infrared high-frequency component to obtain the visible light texture dominant path feature and the infrared intensity dominant path feature; the features of the two paths are aggregated to obtain the shallow fusion feature.

[0011] In at least one embodiment, the semantically guided cross-attention fusion strategy specifically includes: Projecting the deep features of the input generates query vector, key vector, and value vector; Retrieve visible light features using query vectors of infrared features, and retrieve infrared features using query vectors of visible light features; During the fine reasoning stage, spatial attention masks are used to enhance the key vector and value vector; Cross-attention weights are calculated based on the enhanced key vectors and value vectors, features are updated, and the updated features from the two modalities are aggregated into deep fusion features.

[0012] In at least one embodiment, a progressive cascaded network is trained based on a task decoupling meta-optimization strategy, and meta-learning-based two-layer optimization is performed alternately on the fusion task and the segmentation task, respectively, while the shared feature encoder is frozen.

[0013] In at least one embodiment, the task decoupling meta-optimization strategy specifically includes: The network parameter set is defined as including shared feature encoder parameters, fusion decoder parameters, and segmentation decoder parameters; In odd-numbered training epochs, the parameters of the shared feature encoder and the segment decoder are frozen, and only the parameters of the fusion decoder are subjected to a two-layer optimization based on meta-learning. In even-numbered training epochs, the parameters of the shared feature encoder and the fusion decoder are frozen, and only the segment decoder parameters are subjected to a meta-learning-based two-layer optimization.

[0014] In at least one embodiment, the meta-learning-based two-layer optimization includes an inner loop and an outer loop. In the inner loop, the task loss is calculated based on the meta-training set, and a virtual gradient descent step is performed on the decoder parameters corresponding to the task to obtain virtual parameters. In the outer loop, the meta-loss is calculated based on the meta-test set and the virtual parameters, and the gradient of the meta-loss with respect to the original decoder parameters is calculated as the meta-gradient. The meta-gradient is then used to perform the actual update on the original decoder parameters.

[0015] In at least one embodiment, the total loss function of the progressive cascaded network includes fusion loss and segmentation loss; Among them, the fusion loss is a weighted sum of the intensity loss, structural similarity loss, gradient loss and color loss; the segmentation loss is a combination of the cross-entropy loss and the Focal loss.

[0016] Secondly, the technical solution of the present invention also provides a progressive image fusion system based on meta-learning, comprising: The image acquisition module is configured to acquire infrared images and visible light images; The coarse perception and fine reasoning module is configured to: input infrared and visible light images into a progressive cascaded network for image fusion and semantic segmentation; wherein, the progressive cascaded network includes a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series; perform coarse perception on the input image based on the first-level perceptual dual-domain fusion network to generate a preliminary fused image and a coarse semantic segmentation map, and generate a spatial attention mask based on the coarse semantic segmentation map; and perform fine reasoning on the input image based on the spatial attention mask and the second-level perceptual dual-domain fusion network to generate a fine fused image and a fine semantic segmentation map; The image fusion module is configured to: aggregate the preliminary fused image and the fine fused image to obtain the final fused image; The semantic segmentation module is configured to aggregate the coarse semantic segmentation map and the fine semantic segmentation map to obtain the final semantic segmentation map.

[0017] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) The meta-learning-driven progressive image fusion method of the present invention introduces a progressive cascade architecture from coarse to fine, deconstructing the complex fusion task into two stages: coarse perception and fine reasoning. The semantic segmentation results of the coarse stage are used to generate a spatial attention mask, which is used to guide the input modulation and feature fusion of the fine stage. This achieves semantically guided image reconstruction and effectively solves the problem of balancing texture details and noise suppression.

[0018] 2) The hierarchical perception dual-domain fusion network of the present invention adopts a frequency domain decoupling fusion strategy based on discrete wavelet transform for shallow features and a semantically guided cross-attention fusion strategy for deep features. It solves the technical problems of easy loss of shallow details and difficulty in aligning deep semantics respectively. It can not only accurately reconstruct high-frequency textures and suppress noise in complex scenes, but also effectively align cross-modal semantics through semantic feedback mechanism, thereby improving the quality and semantic consistency of fused images.

[0019] 3) This invention proposes a task decoupling meta-optimization strategy, which isolates multi-task gradient interference in the time dimension through alternating update and parameter freezing mechanism, and combines a meta-learning-based two-layer optimization framework to automatically balance the optimization conflict between fusion task and segmentation task. It can achieve synchronous improvement of multi-task performance without manual adjustment of loss weights, and solves the problem of negative transfer in existing methods. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1This is a schematic diagram of the progressive image fusion method based on meta-learning driven by Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the progressive cascaded network disclosed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure of the hierarchical perception dual-domain fusion module disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] As described in the background section, the purpose of this invention is to overcome the shortcomings of the prior art and provide a progressive image fusion method and system based on meta-learning. By introducing a progressive cascading mechanism, a hierarchical perception dual-domain fusion module, and a task decoupling meta-optimization strategy, the quality of the fused image and the performance of downstream tasks are significantly improved.

[0024] Example 1 In a typical embodiment of the present invention, such as Figures 1 to 3 As shown, this embodiment discloses a progressive image fusion method driven by meta-learning, which specifically includes the following steps: S1. Acquire infrared and visible light images; S2. The infrared image and the visible light image are input into a progressive cascaded network for image fusion and semantic segmentation processing; wherein, the progressive cascaded network includes a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series; based on the first-level perceptual dual-domain fusion network, the input image is coarsely perceived to generate a preliminary fused image and a coarse semantic segmentation map, and a spatial attention mask is generated based on the coarse semantic segmentation map; based on the spatial attention mask and the second-level perceptual dual-domain fusion network, the input image is finely reasoned to generate a finely fused image and a finely segmented semantic map; S3. Combine the preliminary fused image and the fine fused image to obtain the final fused image, and combine the coarse semantic segmentation map and the fine semantic segmentation map to obtain the final semantic segmentation map.

[0025] The above-mentioned progressive image fusion method based on meta-learning will be described in detail below with reference to specific implementation methods.

[0026] S1. Acquire infrared and visible light images.

[0027] In this step, infrared images With visible light images Images can be obtained from publicly available data sources or acquired by image acquisition devices; no specific restrictions are imposed here. After acquiring infrared and visible light images, preprocessing operations are performed, including: For infrared images, ensure they are in single-channel grayscale format; for visible light images, maintain their original RGB three-channel structure. Both infrared and visible light images are cropped to 640p. The image size is 480. Subsequently, both types of images are normalized, with pixel values ​​standardized to the [0,1] range and converted to tensor format.

[0028] S2. Input the infrared image and the visible light image into a progressive cascaded network for image fusion and semantic segmentation processing.

[0029] To address the limitations of existing fusion methods based on single-stage reconstruction paradigms, which are constrained by network structure and struggle to simultaneously achieve faithful reconstruction of texture details and effective noise suppression in complex scenes, resulting in fused images prone to edge blurring, insufficient contrast, or artifact retention, this embodiment constructs a progressive cascaded network comprising a coarse perception stage and a fine inference stage. This progressive cascaded network includes a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series, respectively realizing coarse perception and fine inference of the input image.

[0030] like Figure 2 As shown, the progressive cascaded network consists of a coarse perception stage and a fine reasoning stage connected in series. In the coarse perception stage, the infrared image is... With visible light images Input shared encoder First multi-scale features are extracted, and these first multi-scale features are processed by the first-level perceptual dual-domain fusion module, and then a preliminary fused image is generated by the first fusion decoder. A coarse semantic segmentation map is generated using the first segmentation decoder. This stage aims to establish a global understanding of the scene's structure.

[0031] To guide the network to focus on weak targets and key regions, this embodiment utilizes the segmentation results from the coarse perception stage—the coarse semantic segmentation map. Generate spatial attention mask Specifically, firstly, the coarse semantic segmentation map is processed along the channel dimension. Max pooling is performed to extract the class activation values ​​with the strongest responses at each pixel location. Then, spatial context information is aggregated through large kernel convolution to generate a spatial attention mask. Specifically, it is expressed as:

[0032] In the formula, This represents the Sigmoid activation function, used to map the output to... interval; Indicates the kernel size as Convolutional layers are used to capture local spatial context; This represents the max pooling operation. The resulting spatial attention mask. It explicitly indicates semantically salient regions.

[0033] In the fine reasoning stage, the first step is to utilize the generative spatial attention mask. For the original input infrared image With visible light images Semantic refocusing modulation is performed to generate the modulated input image, specifically as follows:

[0034] In the formula, This represents the modulated input image; Represents the original input infrared image or visible light image ; This indicates element-wise multiplication.

[0035] Then, the modulated input image The image is fed into the second-level perceptual dual-domain fusion network. While maintaining the shared encoder parameters, the shared encoder extracts second-level multi-scale features. These second-level multi-scale features are then processed by the second-level perceptual dual-domain fusion module and passed through the second fusion decoder to generate a finely fused image. And a fine semantic segmentation map is generated through the second segmentation decoder. .

[0036] In the feature fusion stage, existing methods often overlook the hierarchical differences in features. Shallow features in deep neural networks mainly encode high-frequency information such as edges and textures, while deep features are rich in abstract information such as semantics and categories. Traditional methods employ a homogeneous fusion strategy for all levels, which has significant drawbacks: in shallow layers, the low-pass filtering properties of spatial domain convolution easily smooth out high-frequency details; in deep layers, due to the inherent bias in semantic representation between infrared and visible light modalities, direct fusion makes it difficult to achieve accurate cross-modal semantic alignment.

[0037] To address this issue, this embodiment implements frequency domain decoupling fusion and semantically guided attention fusion respectively when building the hierarchical perception dual-domain fusion module to address the differences in physical attributes between shallow and deep features.

[0038] As an exemplary implementation, such as Figure 2As shown, the shared encoder consists of four Transformer layers, extracting four scales. The first two Transformer layers extract shallow features, and the last two Transformer layers extract deep features. To address the issue that shallow features, which mainly encode high-frequency information such as edges and textures, are easily affected by the low-pass filtering characteristics of spatial convolution, leading to smoother details, this embodiment designs a spectral decoupling strategy based on discrete wavelet transform. For example... Figure 3 As shown, two-dimensional wavelets are used to extract shallow features from the shared encoder. and Decomposed into low-frequency structural components With high-frequency texture components For the low-frequency structural components of shallow features To adaptively integrate the substrate structure information, a spatial gating fusion mechanism is adopted, which concatenates the low-frequency structural components of the two modalities along the channels and generates adaptive weights through a convolutional network.

[0039] In the formula, Indicates batch normalization; For activation functions; For the Sigmoid function; This indicates splicing along the channel dimension; Represents the adaptive weights for visible light modes; Indicates infrared modal adaptive weights; Low-frequency structural components representing shallow visible light characteristics; Low-frequency structural components representing shallow infrared features.

[0040] The low-frequency structural components are fused based on adaptive weights of two modes to obtain the fused low-frequency features. Specifically, it can be expressed as:

[0041] In the formula, This indicates the low-frequency characteristics after fusion.

[0042] For the high-frequency texture components of shallow features To maximize the preservation of texture details, grouped depthwise convolution (GDConv) is used to enhance the features of high-frequency texture components in infrared and visible light respectively, in order to capture fine gradient information.

[0043] Considering that single-path reconstruction is difficult to balance infrared thermal target intensity and visible light texture clarity, this embodiment introduces a bidirectional reconstruction mechanism, including two dominant paths: a visible light texture dominant path and an infrared intensity dominant path.

[0044] Among them, the dominant path of visible light texture is to fuse the low-frequency features. Combined with the enhanced visible light high-frequency texture components, and reconstructed through inverse wavelet transform, the dominant visible light features are generated. Specifically, it is expressed as:

[0045] In the formula, This indicates inverse wavelet transform reconstruction; Represents grouped depthwise convolution. Represents the high-frequency texture components of visible light. This represents the enhanced visible light high-frequency texture component.

[0046] The infrared intensity-dominant path is to fuse the low-frequency features. Combined with the enhanced infrared high-frequency texture components, infrared dominant features are generated through inverse wavelet transform reconstruction. Specifically, it can be expressed as:

[0047] In the formula, This represents the infrared high-frequency texture component.

[0048] Finally, through The convolutional aggregation yields a shallow fusion feature from the bidirectional results, specifically represented as follows:

[0049] In the formula, This indicates shallow fusion characteristics.

[0050] To address the issue that deep features are rich in semantics but have low spatial resolution, and that direct fusion can easily lead to semantic misalignment, this embodiment designs a semantically guided cross-attention mechanism. For example... Figure 3 As shown, firstly, group normalization (GN) and convolutional projection are performed on the input deep features to generate queries in two modalities. ,key Sum A matrix, specifically represented as:

[0051] In the formula, Indicates the mode corresponding to the feature. Represents visible light modes, Indicates the infrared mode.

[0052] To achieve cross-modal information complementarity, a symmetric query-exchange mechanism is adopted: the visible light branch utilizes infrared query. Retrieve relevant information from visible light features, while the infrared branch uses visible light for querying. Capture prominent areas in infrared signatures.

[0053] Specifically, in the fine reasoning stage, to address the target focusing problem in complex contexts, a spatial attention mask generated based on the coarse perception stage is utilized. t-key Sum Explicit matrix enhancement is specifically represented as follows:

[0054] In the formula, This represents the explicitly enhanced bond matrix; This represents the value matrix after explicit enhancement. This term is used to significantly enhance the weight of the foreground semantic region while preserving the original features, forcing the network to focus on key targets.

[0055] Based on the enhanced key-value pairs, an attention map is calculated and the features are updated, specifically as follows:

[0056]

[0057] In the formula, The dimension of the key vector; Indicates infrared mode update characteristics; This represents the visible light mode update characteristics.

[0058] Finally, the updated features from the two modalities are concatenated and aggregated through convolution to form a deep fusion feature. .

[0059] In fusion scenarios for advanced vision tasks, existing multi-task joint training suffers from significant optimization imbalances: fusion tasks prioritize pixel-level reconstruction, and their loss functions are often orders of magnitude larger than those of segmentation tasks, which prioritize class discrimination. This order-of-magnitude difference leads to the fusion task dominating the parameter update direction of the shared encoder during backpropagation with large gradients. This makes it difficult for the model to learn discriminative features beneficial to downstream tasks, resulting in "negative transfer." Furthermore, existing weighted summation methods rely on manual parameter tuning, making it difficult to adapt to dynamic changes during training and limiting the model's generalization ability.

[0060] To address this issue, this embodiment introduces a task decoupling meta-optimization strategy, which models multi-task training as a two-layer optimization problem. Alternatingly, under the condition of freezing the shared feature encoder, meta-learning-based two-layer optimization is performed on the fusion task and the segmentation task respectively, thus solving the gradient conflict and imbalance of optimization dominance caused by the difference in loss magnitude.

[0061] During the multi-task joint training process, infrared and visible light image datasets covering a variety of complex scenes are first collected, a multimodal dataset with pixel-level semantic annotations is constructed, and image preprocessing operations and the division of training and test sets are performed.

[0062] To fully evaluate the effectiveness and generalization ability of the model in various complex scenarios, a variety of datasets were collected from different public data sources. These included infrared and visible light image pairs under various environments, covering complex urban traffic scenarios, multiple target scenarios under different lighting conditions, and extreme challenging scenarios including occlusion, smoke, and strong light, to ensure the diversity and representativeness of the datasets.

[0063] When dividing the training and testing sets, the MSRS and FMB datasets were selected as the core training and evaluation data. Simultaneously, the M3FD dataset was used for quantitative evaluation of the object detection task. Furthermore, to verify the method's cross-scene generalization ability, the LLVIP dataset was used for generalization testing.

[0064] To address the issue that the fusion loss dominates the gradient update direction in multi-task joint training, making it difficult for the shared encoder to learn discriminative semantic features, the network parameter set is first defined as follows:

[0065] In the formula, Indicates a shared feature encoder; These represent the parameters of the fusion decoder; This represents the parameters of the segmented decoding head.

[0066] To cut off interference paths transmitted by shared parameters, the shared feature encoder is frozen during the optimization phase. It employs an alternating update mechanism: optimizing the fusion task in odd epochs and optimizing the splitting task in even epochs, thereby achieving gradient isolation between tasks in the time dimension.

[0067] The two-layer optimization consists of an inner loop and an outer loop. In the inner loop, virtual parameters are updated based on the meta-training set. In the outer loop, meta-gradients are calculated based on the meta-test set and virtual parameters, and the real parameters are updated.

[0068] Taking the fusion task as an example, in the inner loop phase, the training data of the current batch is divided into a meta-training set. Heyuan Test Set In the inner loop, using Calculate the fusion loss and apply it to the fusion decoder header parameters. Perform one step of virtual gradient descent to obtain temporary virtual parameters. Specifically, it is expressed as:

[0069] In the formula, Indicates the inner learning rate; This represents taking the partial derivative with respect to the parameters of the fused decoder header; This represents the loss function for the fusion task. It's important to note that the temporary virtual parameters obtained in this step... It is only used for subsequent calculations and does not actually update the weights in the network.

[0070] In the outer loop phase, the meta-test set is used. Temporary virtual parameters Based on this, a meta-loss is calculated to quantify the generalization error of the current optimization direction on unknown data. Subsequently, this meta-loss is calculated against the original fused decoder head parameters. The gradient is used to obtain the meta-gradient. Based on meta-gradient Performing an actual update is specifically represented as follows:

[0071]

[0072] In the formula, Indicates the outer learning rate; This represents the complete parameter set containing the virtual fusion decoding header parameters; and These represent the fused decoding header parameters before and after the update, respectively. Through this second-order optimization mechanism, the model can find the parameter update direction that makes the generalization performance of each subtask on unseen data better.

[0073] The meta-update of the segmentation task follows the same two-level optimization logic as the fusion task. Within the update cycle of the segmentation task, the segmentation loss is utilized. Calculate virtual segmentation decoder head parameters And update the original segmentation decoder head parameters based on the meta-gradient. Through this alternating two-layer meta-update, this method can automatically balance the optimization conflicts between fusion and segmentation tasks without requiring manual adjustment of loss weights.

[0074] S3. Combine the preliminary fused image and the fine fused image to obtain the final fused image, and combine the coarse semantic segmentation map and the fine semantic segmentation map to obtain the final semantic segmentation map.

[0075] In this step, a weighted average method is used to combine the preliminary fused image and the fine fused image to obtain the final fused image, specifically as follows:

[0076] In the formula, This represents the final merged image.

[0077] The coarse semantic segmentation map and the fine semantic segmentation map are aggregated by direct summation to obtain the final semantic segmentation map, as shown below:

[0078] In the formula, Final semantic segmentation graph.

[0079] This embodiment uses fusion loss. With segmentation loss The composite loss function is used, and differentiated constraint strategies are formulated according to different stages of coarse perception and fine reasoning to ensure that the output has visual effects, detail accuracy and semantic consistency.

[0080] Considering that progressive cascaded networks adopt a cascaded architecture, this embodiment proposes a phased loss constraint strategy: in the coarse perception stage, in order to quickly establish basic multi-task collaborative capabilities, only segmentation loss and simplified fusion loss (a combination of gradient loss and intensity loss) are used for constraint; in the fine inference stage, in order to refine image quality, segmentation loss and full fusion loss are used for constraint.

[0081] To preserve multi-dimensional source image information in the fusion task, the complete fusion loss is defined as a weighted sum of four sub-losses, specifically expressed as:

[0082] In the formula, Indicates strength loss; Represents structural similarity loss; Indicates gradient loss; Indicates color loss; , , and These are the balance coefficients for each loss term.

[0083] Among them, strength loss This is an intensity constraint based on a maximum selection strategy, constructed to adaptively preserve infrared thermal radiation targets and visible light salient regions at the pixel intensity level. Specifically, it is expressed as:

[0084] In the formula, and Indicates the height and width of the image; This indicates the operation of taking the maximum value pixel by pixel. Describing the L1 norm, For the final merged image.

[0085] Structural similarity loss This is to constrain the consistency of the local structure, specifically expressed as:

[0086] In the formula, Indicates the structural similarity index; superscript This indicates grayscale processing.

[0087] Gradient loss This is to prevent texture smoothing during the fusion process and to forcibly constrain the geometric texture details of the fused image, specifically as follows:

[0088] In the formula, This represents the Sobel gradient operator; This represents the absolute value of the gradient.

[0089] Color loss This constraint, imposed in the YCbCr space, is designed to maintain the naturalness of the fused image when the infrared image lacks color information. Specifically, it is expressed as follows:

[0090] In the formula, superscript and These represent the blue and red chromaticity components, respectively.

[0091] Considering the severe class imbalance problem where background pixels far outnumber target pixels in the scene, the segmentation loss of the segmentation task... The loss function is a combination of cross-entropy loss and Focal loss, specifically expressed as:

[0092] In the formula, Represents cross-entropy loss; This indicates Focal loss.

[0093] The cross-entropy loss is used to measure the global difference between the predicted probability distribution and the true label distribution, and can be specifically expressed as:

[0094] In the formula, Indicates the total number of semantic categories; Represents the spatial coordinate index of a pixel; One-hot encoding of the real label, if the pixel belongs to the category The value is 1 if it is 1, otherwise it is 0. This indicates that the network predicts the pixel belongs to a certain category. The probability of.

[0095] Focal loss is used to reweight the gradient contributions of hard-to-classify samples, and can be specifically expressed as:

[0096] In the formula, The predicted probability corresponding to the true category; As a category balance factor; For focusing parameters. When At that time, this factor will reduce the number of easily classified samples (i.e., The relatively large weights force the network to focus its optimization efforts on difficult samples and minority classes (such as small targets at a distance) during training, thereby improving the robustness of semantic segmentation.

[0097] Example 2 In a typical embodiment of the present invention, this embodiment discloses a progressive image fusion system driven by meta-learning, specifically including: The image acquisition module is configured to acquire infrared images and visible light images; The coarse perception and fine reasoning module is configured to: input infrared and visible light images into a progressive cascaded network for image fusion and semantic segmentation; wherein, the progressive cascaded network includes a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series; perform coarse perception on the input image based on the first-level perceptual dual-domain fusion network to generate a preliminary fused image and a coarse semantic segmentation map, and generate a spatial attention mask based on the coarse semantic segmentation map; and perform fine reasoning on the input image based on the spatial attention mask and the second-level perceptual dual-domain fusion network to generate a fine fused image and a fine semantic segmentation map; The image fusion module is configured to: aggregate the preliminary fused image and the fine fused image to obtain the final fused image; The semantic segmentation module is configured to aggregate the coarse semantic segmentation map and the fine semantic segmentation map to obtain the final semantic segmentation map.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A progressive image fusion method based on meta-learning driven by a method characterized by, include: Acquire infrared and visible light images; Infrared and visible light images are input into a progressive cascaded network for image fusion and semantic segmentation. The progressive cascaded network comprises a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series. The first-level perceptual dual-domain fusion network performs coarse perception on the input image, generating a preliminary fused image and a coarse semantic segmentation map. A spatial attention mask is then generated based on the coarse semantic segmentation map. Finally, the spatial attention mask and the second-level perceptual dual-domain fusion network perform fine reasoning on the input image, generating a fine fused image and a fine semantic segmentation map. The initial fused image and the fine fused image are combined to obtain the final fused image, and the coarse semantic segmentation image and the fine semantic segmentation image are combined to obtain the final semantic segmentation image.

2. The progressive image fusion method based on meta-learning as described in claim 1, characterized in that, The method for generating a spatial attention mask based on a coarse semantic segmentation map includes: performing max pooling along the channel dimension of the coarse semantic segmentation map to extract the class activation value with the strongest response at each pixel position; performing large kernel convolution on the pooled result to aggregate spatial context information; and mapping the output to the [0,1] interval through the Sigmoid activation function to generate a spatial attention mask.

3. The progressive image fusion method based on meta-learning as described in claim 1, characterized in that, The first-level perceptual dual-domain fusion network and the second-level perceptual dual-domain fusion network employ different fusion strategies for shallow and deep features. Specifically, for shallow features, a frequency domain decoupling fusion strategy based on discrete wavelet transform is adopted; for deep features, a semantically guided cross-attention fusion strategy is adopted.

4. The progressive image fusion method based on meta-learning as described in claim 3, characterized in that, Frequency domain decoupling and fusion strategies based on discrete wavelet transform specifically include: Two-dimensional discrete wavelet transform is used to decompose the input shallow features into low-frequency structural components and high-frequency texture components; For the low-frequency components of infrared and visible light images, an adaptive weight fusion is performed using a spatially gated fusion module; For the high-frequency components of infrared and visible light images, grouped depthwise convolution is used for feature enhancement respectively; The fused low-frequency component is reconstructed by inverse wavelet transform with the enhanced visible light high-frequency component and infrared high-frequency component to obtain the visible light texture dominant path feature and the infrared intensity dominant path feature; the features of the two paths are aggregated to obtain the shallow fusion feature.

5. The progressive image fusion method based on meta-learning as described in claim 3, characterized in that, Semantic-guided cross-attention fusion strategies specifically include: Projecting the deep features of the input generates query vector, key vector, and value vector; Retrieve visible light features using query vectors of infrared features, and retrieve infrared features using query vectors of visible light features; During the fine reasoning stage, the spatial attention mask is used to enhance the key vector and value vector; Cross-attention weights are calculated based on the enhanced key vectors and value vectors, features are updated, and the updated features from the two modalities are aggregated into deep fusion features.

6. The progressive image fusion method based on meta-learning as described in claim 1, characterized in that, A progressive cascaded network is trained based on a task decoupling meta-optimization strategy. Alternatingly, under the condition of freezing the shared feature encoder, a meta-learning-based two-layer optimization is performed on the fusion task and the segmentation task respectively.

7. The progressive image fusion method based on meta-learning as described in claim 6, characterized in that, The specific optimization strategies for task decoupling include: The network parameter set is defined as including shared feature encoder parameters, fusion decoder parameters, and segmentation decoder parameters; In odd-numbered training epochs, the parameters of the shared feature encoder and the segment decoder are frozen, and only the parameters of the fusion decoder are subjected to a two-layer optimization based on meta-learning. In even-numbered training epochs, the parameters of the shared feature encoder and the fusion decoder are frozen, and only the segment decoder parameters are subjected to a meta-learning-based two-layer optimization.

8. The progressive image fusion method based on meta-learning as described in claim 6, characterized in that, The meta-learning-based two-layer optimization includes an inner loop and an outer loop. In the inner loop, the task loss is calculated based on the meta-training set, and a virtual gradient descent step is performed on the decoder parameters corresponding to the task to obtain virtual parameters. In the outer loop, the meta-loss is calculated based on the meta-test set and virtual parameters, and the gradient of the meta-loss with respect to the original decoder parameters is calculated as the meta-gradient. The meta-gradient is then used to perform the actual update of the original decoder parameters.

9. The progressive image fusion method based on meta-learning as described in claim 1, characterized in that, The total loss function of a progressively cascaded network includes fusion loss and segmentation loss; Among them, the fusion loss is a weighted sum of the intensity loss, structural similarity loss, gradient loss and color loss; the segmentation loss is a combination of the cross-entropy loss and the Focal loss.

10. A progressive image fusion system based on meta-learning, characterized in that, include: The image acquisition module is configured to acquire infrared images and visible light images; The coarse perception and fine reasoning module is configured to: input infrared and visible light images into a progressive cascaded network for image fusion and semantic segmentation; wherein, the progressive cascaded network includes a first-level perceptual dual-domain fusion network and a second-level perceptual dual-domain fusion network connected in series; perform coarse perception on the input image based on the first-level perceptual dual-domain fusion network to generate a preliminary fused image and a coarse semantic segmentation map, and generate a spatial attention mask based on the coarse semantic segmentation map; and perform fine reasoning on the input image based on the spatial attention mask and the second-level perceptual dual-domain fusion network to generate a fine fused image and a fine semantic segmentation map; The image fusion module is configured to: aggregate the preliminary fused image and the fine fused image to obtain the final fused image; The semantic segmentation module is configured to aggregate the coarse semantic segmentation map and the fine semantic segmentation map to obtain the final semantic segmentation map.

Images