Robust image fusion method and system based on visual semantic collaboration

Abstract

The invention provides a robust image fusion method and system based on visual semantic collaboration, the difference between visual information and semantic information is bridged through a feature search module, and the consistency of task learning is ensured by using a gradient correction strategy; a potential basis for mutual enhancement of visual fusion and segmentation is established, target visibility in a fused image is enhanced by using semantic-guided fusion loss, and understanding and decision making of semantic segmentation on a degraded scene are improved through information recovery prior embedding; a multi-modal information interaction mechanism is designed, an interactive gating mixed attention module and a progressive coupling attention module are developed, the problem that feature validity scores are not matched is solved, the feature interaction and aggregation capability is further improved, and high-quality information guarantee is provided for vision-semantic collaboration.

Description

Technical Field

[0001] This invention relates to the field of computer vision image processing technology, and in particular to a robust image fusion method and system based on visual semantic collaboration. Background Technology

[0002] Due to the inherent limitations of imaging principles, a single sensor can only capture information relevant to specific aspects of the imaged scene. In other words, it cannot support the comprehensive scene representation requirements, especially under adverse conditions. Therefore, this not only hinders visual perception for human users but also makes semantic understanding and decision-making more difficult for machines. Against this backdrop, multimodal image fusion technology has emerged, enhancing visual and semantic perception capabilities by combining image data captured by various sensors. To date, this technology has been widely applied in various fields, such as security monitoring and nighttime driving assistance, demonstrating significant value.

[0003] According to the technical definition, the goals of multimodal image fusion include both visual and semantic aspects, which gives rise to two specific fusion tasks: 1) Visual fusion: aims to generate a single fused image with rich texture details and significant contrast to improve visual perception. 2) Semantic fusion: focuses on enhancing machine understanding capabilities and strengthening decision-making and description of imaging scenes, such as semantic segmentation and object detection. These two tasks form a complete perception enhancement system from the visual to the semantic levels.

[0004] However, a key limitation remains: existing methods do not adequately consider the intrinsic relationship between visual and semantic fusion. This leads to a significant optimization gap between the visual and semantic domains in perceptual augmentation systems, hindering further performance improvements. Summary of the Invention

[0005] This invention provides a robust image fusion method and system based on visual-semantic collaboration, which addresses the shortcomings of existing technologies that cannot fully consider the intrinsic relationship between visual fusion and semantic fusion, and realizes an image fusion method based on visual-semantic collaboration.

[0006] This invention provides a robust image fusion method based on visual semantic collaboration, comprising: Visible light images and infrared images are input into a pre-trained multi-task learning model, and visual fusion features and semantic fusion features are extracted by the encoding modules of the visual fusion branch and semantic fusion branch of the multi-task learning model, respectively. The visual fusion features and the semantic fusion features are integrated in the feature search module of the multi-task learning model to generate a shared feature pool, and the enhanced visual fusion features and enhanced semantic fusion features are obtained based on the shared feature pool. The enhanced visual fusion features and enhanced semantic fusion features are respectively input into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch to obtain the visual fusion image and the semantic fusion image; The loss function used by the visual fusion branch during the training process is used to make the output image of the visual fusion branch highlight the salient features of the input image in the thermal radiation region, and drive the fusion result to be smoother in the non-thermal radiation region.

[0007] According to the present invention, a robust image fusion method based on visual semantic collaboration is provided, wherein the encoding module is composed of multiple sequentially connected encoding network layers, and each encoding network layer includes a cascaded interactive gated hybrid attention module and a progressively coupled attention module; The interactive gated hybrid attention module is used to perform local cross-modal interaction of visible light features and infrared features. After correcting misjudgments in pooling attention through a gating mechanism, the visible light features and infrared features are updated. The visible light features and infrared features of the first-level coding network layer are extracted from the input visible light image and infrared image, while the visible light features and infrared features of the remaining coding network layers are the outputs of the interactive gated hybrid attention module in the previous coding network layer. The progressively coupled attention module performs cross-modal global integration of the updated visible light and infrared features and outputs the depth features corresponding to the coding network layer. The output of the encoding module is composed of the deep features output by the progressively coupled attention modules of each level of the encoding network layer.

[0008] According to the robust image fusion method based on visual-semantic collaboration provided by the present invention, the visual fusion features and the semantic fusion features are integrated in the feature search module of the multi-task learning model to generate a shared feature pool, and the enhanced visual fusion features and enhanced semantic fusion features are obtained based on the shared feature pool. The specific steps include: The visual fusion features and the semantic fusion features are integrated to generate a shared feature pool; After the visual fusion feature is concatenated with the shared feature pool, it is processed by a 1-0 convolution to generate an enhanced visual fusion feature. The semantic fusion feature is concatenated with the shared feature pool and then processed by a 1-0 convolution to generate an enhanced semantic fusion feature.

[0009] According to the robust image fusion method based on visual semantic collaboration provided by the present invention, the loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the input visible light image and infrared image. In non-thermal radiation regions, the fusion result driven by the visual fusion branch approaches the weighted average of the pixel intensities at corresponding positions in the input visible light image and infrared image.

[0010] According to the robust image fusion method based on visual semantic collaboration provided by the present invention, the loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the visible light optimized image and the infrared optimized image. In non-thermal radiation regions, the fusion result driving the visual fusion branch output approaches the weighted average of the pixel intensities at corresponding positions in the visible light optimized image and the infrared optimized image. The visible light optimized image and the infrared optimized image are clean images obtained by image enhancement processing of the input visible light image and infrared image, respectively.

[0011] According to the present invention, a robust image fusion method based on visual semantic collaboration is provided, wherein the multi-task learning model uses a gradient correction strategy during training, the gradient correction strategy including gradient conflict correction and gradient balancing aggregation; The gradient conflict correction is used to randomly select the gradient of one task and project it onto the normal plane of the gradient of the other task when the angle between the gradients of the visual fusion task and the semantic fusion task is an obtuse angle. The gradient balancing aggregation is used to aggregate weighted gradients into shared gradients for parameter updates in multi-task learning models.

[0012] This invention also provides a robust image fusion system based on visual semantic collaboration, comprising: The acquisition module is used to acquire visible light images and infrared images; The dual-branch extraction module is used to extract visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and the semantic fusion branch, respectively. The feature search module is used to integrate the visual fusion features and the semantic fusion features to generate a shared feature pool, and to obtain enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; The dual-branch output module is used to input the enhanced visual fusion features and the enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain the visual fusion image and the semantic fusion image.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robust image fusion method based on visual semantic collaboration as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robust image fusion method based on visual semantic collaboration as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robust image fusion method based on visual semantic collaboration as described above.

[0016] The robust image fusion method and system based on visual-semantic collaboration provided by this invention bridges the gap between visual and semantic information through a feature search module, establishes a potential foundation for mutual reinforcement between visual fusion and segmentation through a designed loss function, enhances the visibility of targets in the fused image by utilizing semantically guided fusion loss, and achieves synergistic optimization of visual fusion and semantic fusion through a multimodal information interaction mechanism. This improves the visibility and distinguishability of the fused image under complex degradation environments, as well as the accuracy and robustness of target segmentation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 2 This is a schematic diagram of the visual fusion branch and semantic fusion branch encoding modules in the robust image fusion method based on visual-semantic collaboration provided by the present invention; Figure 3 This is one of the schematic diagrams of the feature search module in the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 4 This is the second schematic diagram of the feature search module in the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 5This is a schematic diagram illustrating the underlying basis of the visual-semantic mutual reinforcement in the robust image fusion method based on visual-semantic collaboration provided by the present invention. Figure 6 This is a schematic diagram of the gradient correction strategy used in the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 7 This is a comparative diagram of visual fusion image results obtained by the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 8 This is a comparative diagram of semantic fusion image results obtained by the robust image fusion method based on visual semantic collaboration provided by the present invention; Figure 9 This is a schematic diagram of the structure of the robust image fusion system based on visual semantic collaboration provided by the present invention; Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Specifically, based on the relationship between visual fusion and semantic fusion tasks, existing multimodal image fusion methods can be divided into two paradigms: 1) Independent optimization paradigm: Visual fusion and semantic fusion are generally considered to be two independent processes. Therefore, these methods often design methods specifically for the characteristics of these two tasks, which means that these methods waste the prior knowledge provided by each task for the other during the model solving process, leading to inevitable performance bottlenecks for both. 2) Sequential iterative optimization paradigm: Semantic fusion is considered to be the next step after visual fusion, and the two should strictly satisfy a sequential iterative relationship. However, visual fusion is essentially an information compression process, which reduces the amount of information available for semantic fusion, thus negatively impacting the final decision. Furthermore, inaccurate semantic decisions, in turn, affect the optimization of visual fusion, resulting in poor performance of the entire perception enhancement system. In summary, exploring the collaborative relationship between visual and semantic fusion and constructing a more intelligent mutual reinforcement mechanism is key to improving perception capabilities. To this end, this invention provides a robust image fusion algorithm based on visual-semantic collaboration.

[0021] The following is combined Figures 1 to 8 This invention introduces a robust image fusion method based on visual semantic collaboration, such as... Figure 1 As shown, it includes: Step 101: Input the visible light image and the infrared image into the pre-trained multi-task learning model, and extract the visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and the semantic fusion branch of the multi-task learning model, respectively. Acquire the registered visible light and infrared images, and input the visible light and infrared images into the pre-trained multi-task learning model.

[0022] The multi-task learning in this invention includes two task branches: a visual fusion branch and a semantic fusion branch, which respectively perform the visual fusion task and the semantic fusion task.

[0023] During the encoding phase, the visual fusion branch and the semantic fusion branch use structurally identical encoding modules to process visible light images, respectively. and infrared images Feature extraction is performed to obtain visual fusion features and semantic fusion features.

[0024] Step 102: The visual fusion feature and the semantic fusion feature are integrated in the feature search module of the multi-task learning model to generate a shared feature pool, and the enhanced visual fusion feature and the enhanced semantic fusion feature are obtained based on the shared feature pool. The extracted visual fusion features and semantic fusion features are then input into the feature search module for integration.

[0025] Specifically, the feature search module, built across network layers of different task encoders, allows extracted visual fusion features and semantic fusion features to search for each other, thereby complementing and enriching the potential of visual and semantic features. From an information capacity perspective, this provides sufficient support for visual and semantic fusion to achieve further optimization and improvement.

[0026] Optionally, visual fusion features and semantic fusion features are first integrated to generate a shared feature pool. Then, the visual fusion features and semantic fusion features are respectively extracted from the shared feature pool to expand themselves, resulting in enhanced visual fusion features and enhanced semantic fusion features.

[0027] Step 103: Input the enhanced visual fusion features and the enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain the visual fusion image and the semantic fusion image; The loss function used by the visual fusion branch during the training process is used to make the output image of the visual fusion branch highlight the salient features of the input image in the thermal radiation region, and drive the fusion result to be smoother in the non-thermal radiation region.

[0028] Based on this, the enhanced visual fusion features and enhanced semantic fusion features are respectively input into the decoding modules of the corresponding branches for decoding, resulting in the decoding module of the visual fusion branch. D f Output visual fusion image I f and the decoding module of the semantic fusion branch. D s Output semantic fusion image I s .

[0029] In the training process of the multi-task learning model, the semantic fusion branch is trained under the constraints of conventional segmentation loss, while the visual fusion branch is trained under the designed loss function.

[0030] Specifically, in order to enhance the visibility of targets in the fused image by utilizing semantically guided fusion loss, the loss function used by the visual fusion branch during training is configured to make the output image of the visual fusion branch highlight the salient features in the input infrared / visible light image in the thermal radiation region, strengthen the key semantic regions represented by the thermal radiation region, and drive the smoothing of the fusion result region in the non-thermal radiation region, so as to better preserve the effectiveness of the image information of the background region represented by the non-thermal radiation region.

[0031] In each training round, the multi-task learning model updates the parameters of the feature search module shared by the two task branches through gradient backpropagation, so as to synergistically improve the overall performance of visual fusion and semantic fusion tasks.

[0032] This invention bridges the gap between visual and semantic information through a feature search module, establishes a potential foundation for mutual reinforcement between visual fusion and segmentation through a designed loss function, enhances the visibility of targets in the fused image by utilizing semantically guided fusion loss, and achieves synergistic optimization of visual fusion and semantic fusion through a multimodal information interaction mechanism. This improves the visibility and distinguishability of the fused image in complex degradation environments, as well as the accuracy and robustness of target segmentation.

[0033] In the robust image fusion method based on visual semantic collaboration of the present invention, the coding module is composed of multiple sequentially connected coding network layers, each coding network layer including a cascaded interactive gated hybrid attention module and a progressively coupled attention module; The interactive gated hybrid attention module is used to perform local cross-modal interaction of visible light features and infrared features. After correcting misjudgments in pooling attention through a gating mechanism, the visible light features and infrared features are updated. The visible light features and infrared features of the first-level coding network layer are extracted from the input visible light image and infrared image, while the visible light features and infrared features of the remaining coding network layers are the outputs of the interactive gated hybrid attention module in the previous coding network layer. The progressively coupled attention module performs cross-modal global integration of the updated visible light and infrared features and outputs the depth features corresponding to the coding network layer. The output of the encoding module is composed of the deep features output by the progressively coupled attention modules of each level of the encoding network layer.

[0034] In this embodiment, the structures of the encoding modules used in the visual fusion branch and the semantic fusion branch are as follows: Figure 2 As shown.

[0035] The encoding module consists of a k-level encoding network, which includes an interactive gated hybrid attention module (IGM-Att) and a progressively coupled attention module (PC-Att) to achieve an effective multimodal information interaction mechanism, providing strong information support for visual fusion and semantic segmentation. Specifically: The first-level coding network also includes a basic feature extraction module, which extracts k basic features from the input visible light image. and infrared images Initial extraction is performed to achieve layer-by-layer feature transfer: ; In the formula, and These represent the k-th basic feature extraction modules on the visible light and infrared branches, respectively. This represents the visible and infrared features refined by the (k-1)th IGM-Att module. When k=1, these features are initialized with the input visible and infrared images. and These represent the visible light features and infrared features extracted by the basic feature extraction module, respectively.

[0036] The basic feature extraction module extracts the following features. and The input is then an interactive gated hybrid attention module, which uses a CNN architecture to achieve local interaction of cross-modal features and combines a gating mechanism to correct misjudgments in traditional pooling attention. ; In the formula, This represents the function for the k-th IGM-Att module. Specifically, IGM-Att uses traditional channel and spatial attention to model the relationships between multimodal features. Channel attention is calculated as follows: ; In the formula, This represents the activation function for the impulse response. MLP For multilayer perceptron functions, AP This indicates the average pooling operation. MP This indicates a max pooling operation. C This indicates a connection along the channel dimension. Spatial attention is calculated as follows: ; In the formula, Conv This represents a convolution operation. The gating mechanism addresses the feature effectiveness score mismatch that may arise from pooling-based attention mechanisms by calculating a gating correction score: ; For mixed weights Further details: ; In the formula, [0] and [1] represent vector indices.

[0037] The above methods improve the accuracy of feature importance assessment, strengthen valuable features, and effectively suppress redundant features. Finally, the obtained weights are used to perform local interactions on the features, updating the input visible light and infrared features: .

[0038] Based on this, the IGM-Att module in each layer of the coding network serves as the input to the next level of the IGM-Att module, and also as the input to the progressively coupled attention module of the coding network at that level.

[0039] The progressively coupled attention module PC-Att in this embodiment adopts the Transformer architecture to achieve global integration of cross-modal features, thereby enhancing the comprehensive understanding of the scene. ; In the formula, This represents the function of the k-th PC-Att module. and represents the output of the k-th PC-Att module in the visual fusion branch and the output of the k-th PC-Att module in the semantic fusion branch, respectively.

[0040] Specifically, two information refinement strategies were adopted, including single-modal self-reinforcement and cross-modal complementary enhancement, and applied in a progressive manner: ; In the formula, Self This indicates computation using a self-attention mechanism. Cross This indicates cross-attention calculation. FL This indicates a flattening operation.

[0041] Based on the encoding module described above, multimodal information interaction is performed independently in the visual fusion branch and the semantic fusion branch to alleviate the feature effectiveness score mismatch problem and further improve feature interaction and aggregation capabilities, providing high-quality information assurance for visual-semantic collaboration. The visual fusion features obtained by the encoder module of the visual fusion branch are the set of outputs from each PC-Att module, i.e. The semantic fusion features extracted by the encoding module of the semantic fusion branch are the set of outputs from each level of the PC-Att module, i.e. .

[0042] In the robust image fusion method based on visual-semantic collaboration of the present invention, the steps of integrating the visual fusion features and the semantic fusion features in the feature search module of the multi-task learning model to generate a shared feature pool, and obtaining enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool, specifically include: The visual fusion features and the semantic fusion features are integrated to generate a shared feature pool; After the visual fusion feature is concatenated with the shared feature pool, it is processed by a 1-0 convolution to generate an enhanced visual fusion feature. The semantic fusion feature is concatenated with the shared feature pool and then processed by a 1-0 convolution to generate an enhanced semantic fusion feature.

[0043] In this embodiment, based on the feature search module, a collaborative optimization between visual fusion and semantic segmentation is further constructed on the basis of establishing a mutual reinforcement and multimodal information interaction mechanism, ensuring the stable progress of mutual reinforcement from both physical connection and optimization perspectives.

[0044] Specifically, such as Figure 3 and Figure 4 As shown, fully integrated and To generate a shared feature pool : ; Then, and It needs to extract features from the shared feature pool to expand itself: ; In the formula, This represents a 1-0 convolution, meaning that at the start of optimization, the convolution kernel is initialized to binary form along the channels. For enhanced visual fusion features, This refers to the enhanced semantic fusion features.

[0045] In the above method, 1-0 convolution is used to set the weight of the original features to 1 and the weight of the searched features to 0. Then, based on dynamic learning, the searched features are supplemented with the original visual fusion features and semantic fusion features in a more stable way.

[0046] In the robust image fusion method based on visual-semantic collaboration of this invention, the loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the input visible light image and infrared image. In non-thermal radiation regions, the fusion result driven by the visual fusion branch approaches the weighted average of the pixel intensities at corresponding positions in the input visible light image and infrared image.

[0047] In this embodiment, a segmentation-guided fusion loss function is designed, using pixel intensity as the basic image representation, and segmentation labels are used to guide the learning strategy for different regions: ; In the formula, arg It is used for splitting tags Central positioning specifies the category of thermal radiation targets. Search operators for a range of pixels. Here, The targets were specifically categorized into two types: "people" and "vehicles". T This represents a selection map used to provide unique fusion guidance strategies for different regions. The segmentation labels are provided by the dataset.

[0048] In the target area of ​​thermal radiation, use max The function selects the most important pixel intensity at the corresponding location as the optimization objective because the importance of these semantic categories is reflected in high pixel values. Conversely, for non-thermal radiation target regions, a dynamically weighted guidance function is developed based on the region effectiveness score.

[0049] Specifically, the local contrast of multiple local regions is calculated and aggregated to represent the information effectiveness of the image. Therefore, the region effectiveness score of a single-modal image is defined as follows: ,in p Represents an image block. SD It is a standard deviation function. Therefore, the weighting factor can be calculated as... ,i Represents a mode.

[0050] The loss function configured in the above manner allows semantic segmentation to participate in the optimization of visual fusion, providing a basis for the mutual promotion of visual fusion and semantic fusion tasks. Its construction basis is as follows: Figure 4 As shown.

[0051] In the robust image fusion method based on visual-semantic collaboration of this invention, the loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the visible light optimized image and the infrared optimized image. In non-thermal radiation regions, the fusion result driving the visual fusion branch output approaches the weighted average of the pixel intensities at corresponding positions in the visible light optimized image and the infrared optimized image. The visible light optimized image and the infrared optimized image are clean images obtained by image enhancement processing of the input visible light image and infrared image, respectively.

[0052] In this embodiment, based on the loss function construction and dataset used in the previous embodiment, image enhancement processing is first performed on the visible light and infrared images in the dataset to obtain corresponding clean images, namely, the visible light optimized image and the infrared optimized image, so as to achieve prior embedding of information recovery through the segmentation-guided fusion loss function: ; In the formula, and This indicates visible light optimized images and infrared optimized images.

[0053] By removing noise, low light, color cast, haze, etc. in visible light images, and low contrast and noise in infrared images, various degradation removals are achieved, thereby giving visual fusion higher robustness and making the positive impact of visual fusion on segmentation clear.

[0054] Based on this, an expert dataset is constructed using visible light optimized images and infrared optimized images obtained through image augmentation. These images are then used in training to improve semantic segmentation's understanding and decision-making regarding degraded scenes through prior embedding of information recovery. This also enables the multi-task learning model trained using the loss function in this implementation to achieve better image fusion performance.

[0055] In the robust image fusion method based on visual semantic collaboration of the present invention, the multi-task learning model uses a gradient correction strategy during training, which includes gradient conflict correction and gradient balancing aggregation. The gradient conflict correction is used to randomly select the gradient of one task and project it onto the normal plane of the gradient of the other task when the angle between the gradients of the visual fusion task and the semantic fusion task is an obtuse angle. The gradient balancing aggregation is used to aggregate weighted gradients into shared gradients for parameter updates in multi-task learning models.

[0056] To further achieve collaborative optimization between the two task branches, this implementation uses a gradient correction strategy during the training of the multi-task learning model to ensure the visual fusion loss is minimized at the optimization level. and semantic segmentation loss The tasks reinforce each other.

[0057] like Figure 6 As shown, when the angle between the gradients of the visual fusion task and the semantic segmentation task is obtuse, it indicates an optimization difference between the two tasks. In this case, the gradient of one task is randomly selected and projected onto the normal plane of the gradient of the other task. ; In the formula, and Representing tasks i and tasks j The gradient.

[0058] The gradient conflict correction strategy described above reduces the risk of mutual damage between visual fusion and semantic segmentation.

[0059] Furthermore, inconsistent learning intensity can also lead to imbalance in non-conflict scenarios. The shared gradient is obtained by calculating the benefit-proportion balance solution for each task using the Nash-MTL method: ; Among them, the coefficients used for aggregating gradients By solving The obtained, of which G yes and The gradient matrix formed by column concatenation.

[0060] It is understandable that the gradient calculation mentioned above is the gradient of the feature search module shared by the two task branches, so that during the training process, the loss of the visual fusion branch and the loss of the semantic fusion branch can be combined in the feature search module through gradient backpropagation, that is, the gradient of these two losses relative to the parameters of the feature search module is calculated.

[0061] Based on the above, a complete multi-task learning model was finally designed and trained. This model achieves mutual promotion between the visual fusion task and the semantic fusion task through both architectural coupling and optimization coupling, resulting in higher quality visual and semantic segmentation results. The comparison of their fusion results is shown below. Figure 7 and Figure 8 As shown.

[0062] In summary, compared to the independent optimization paradigm and sequential iterative paradigm used in existing methods, the former wastes the prior knowledge provided by each task for the other, while the latter is limited by the potential negative impact of visual fusion information compression on the final decision of semantic fusion. Furthermore, the inaccuracy of semantic decision-making in turn affects the optimization of visual fusion. This invention introduces a collaborative optimization method for visual and semantic fusion, bridging the gap between visual and semantic information through a feature search module and ensuring the consistency of task learning through a gradient correction strategy. Secondly, existing methods lack exploration of the potential foundation for mutual reinforcement between visual and semantic fusion. This invention establishes the potential foundation for mutual reinforcement between visual fusion and segmentation, enhancing the visibility of targets in the fused image through semantically guided fusion loss, and improving the understanding and decision-making of degraded scenes through information recovery prior embedding. Finally, existing methods suffer from mismatched feature effectiveness scores and insufficient feature aggregation capabilities when using traditional pooling attention. This invention designs a multimodal information interaction mechanism, developing an interactive gated hybrid attention module and a progressively coupled attention module to alleviate the feature effectiveness score mismatch problem and further improve feature interaction and aggregation capabilities, providing high-quality information assurance for visual-semantic collaboration.

[0063] The robust image fusion system based on visual semantic collaboration provided by the present invention will be described below. The robust image fusion system based on visual semantic collaboration described below can be referred to in correspondence with the robust image fusion method based on visual semantic collaboration described above.

[0064] like Figure 9 As shown, the robust image fusion system based on visual semantic collaboration includes an acquisition module 901, a dual-branch extraction module 902, a feature search module 903, and a dual-branch output module 904. Acquisition module 901 is used to acquire visible light images and infrared images; Acquire the registered visible light and infrared images, and input the visible light and infrared images into the pre-trained multi-task learning model.

[0065] The dual-branch extraction module 902 is used to extract visual fusion features and semantic fusion features respectively through the encoding modules of the visual fusion branch and the semantic fusion branch; The multi-task learning in this invention includes two task branches: a visual fusion branch and a semantic fusion branch, which respectively perform the visual fusion task and the semantic fusion task.

[0066] During the encoding phase, the visual fusion branch and the semantic fusion branch use structurally identical encoding modules to process visible light images, respectively. and infrared images Feature extraction is performed to obtain visual fusion features and semantic fusion features.

[0067] The feature search module 903 is used to integrate the visual fusion features and the semantic fusion features to generate a shared feature pool, and to obtain enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; The extracted visual fusion features and semantic fusion features are then input into the feature search module for integration.

[0068] Specifically, the feature search module, built across network layers of different task encoders, allows extracted visual fusion features and semantic fusion features to search for each other, thereby complementing and enriching the potential of visual and semantic features. From an information capacity perspective, this provides sufficient support for visual and semantic fusion to achieve further optimization and improvement.

[0069] Optionally, visual fusion features and semantic fusion features are first integrated to generate a shared feature pool. Then, the visual fusion features and semantic fusion features are respectively extracted from the shared feature pool to expand themselves, resulting in enhanced visual fusion features and enhanced semantic fusion features.

[0070] The dual-branch output module 904 is used to input the enhanced visual fusion features and the enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain the visual fusion image and the semantic fusion image.

[0071] Based on this, the enhanced visual fusion features and enhanced semantic fusion features are respectively input into the decoding modules of the corresponding branches for decoding, resulting in the decoding module of the visual fusion branch. D f Output visual fusion image I f and the decoding module of the semantic fusion branch. D s Output semantic fusion image I s .

[0072] In the training process of the multi-task learning model, the semantic fusion branch is trained under the constraints of conventional segmentation loss, while the visual fusion branch is trained under the designed loss function.

[0073] Specifically, in order to enhance the visibility of targets in the fused image by utilizing semantically guided fusion loss, the loss function used by the visual fusion branch during training is configured to make the output image of the visual fusion branch highlight the salient features in the input infrared / visible light image in the thermal radiation region, strengthen the key semantic regions represented by the thermal radiation region, and drive the smoothing of the fusion result region in the non-thermal radiation region, so as to better preserve the effectiveness of the image information of the background region represented by the non-thermal radiation region.

[0074] In each training round, the multi-task learning model updates the parameters of the feature search module shared by the two task branches through gradient backpropagation, so as to synergistically improve the overall performance of visual fusion and semantic fusion tasks.

[0075] This invention bridges the gap between visual and semantic information through a feature search module, establishes a potential foundation for mutual reinforcement between visual fusion and segmentation through a designed loss function, enhances the visibility of targets in the fused image by utilizing semantically guided fusion loss, and achieves synergistic optimization of visual fusion and semantic fusion through a multimodal information interaction mechanism. This improves the visibility and distinguishability of the fused image in complex degradation environments, as well as the accuracy and robustness of target segmentation.

[0076] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communications bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a robust image fusion method based on visual-semantic collaboration. This method includes: inputting visible light and infrared images into a pre-trained multi-task learning model; extracting visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and semantic fusion branch of the multi-task learning model, respectively; integrating the visual fusion features and semantic fusion features in the feature search module of the multi-task learning model to generate a shared feature pool; and obtaining enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; inputting the enhanced visual fusion features and enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and semantic fusion branch, respectively, to obtain a visually fused image and a semantically fused image; wherein, the loss function used by the visual fusion branch during training is used to make the output image of the visual fusion branch highlight the salient features of the input image in the thermal radiation region and drive the fusion result to tend to be smooth in the non-thermal radiation region.

[0077] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robust image fusion method based on visual-semantic collaboration provided by the above methods. The method includes: inputting visible light images and infrared images into a pre-trained multi-task learning model; extracting visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and semantic fusion branch of the multi-task learning model, respectively; integrating the visual fusion features and the semantic fusion features in the feature search module of the multi-task learning model to generate a shared feature pool; and obtaining enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; inputting the enhanced visual fusion features and enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain a visual fusion image and a semantic fusion image; wherein, the loss function used by the visual fusion branch during the training process is used to make the image output by the visual fusion branch highlight the salient features of the input image in the thermal radiation region and drive the fusion result to tend to be smooth in the non-thermal radiation region.

[0079] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the robust image fusion method based on visual-semantic collaboration provided by the above methods. The method includes: inputting visible light images and infrared images into a pre-trained multi-task learning model; extracting visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and semantic fusion branch of the multi-task learning model, respectively; integrating the visual fusion features and the semantic fusion features in the feature search module of the multi-task learning model to generate a shared feature pool; and obtaining enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; inputting the enhanced visual fusion features and enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain a visual fusion image and a semantic fusion image; wherein, the loss function used by the visual fusion branch during the training process is used to make the image output by the visual fusion branch highlight the salient features of the input image in the thermal radiation region and drive the fusion result to tend to be smooth in the non-thermal radiation region.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust image fusion method based on visual-semantic collaboration, characterized in that, include: Visible light images and infrared images are input into a pre-trained multi-task learning model, and visual fusion features and semantic fusion features are extracted by the encoding modules of the visual fusion branch and semantic fusion branch of the multi-task learning model, respectively. The visual fusion features and the semantic fusion features are integrated in the feature search module of the multi-task learning model to generate a shared feature pool, and the enhanced visual fusion features and enhanced semantic fusion features are obtained based on the shared feature pool. The enhanced visual fusion features and enhanced semantic fusion features are respectively input into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch to obtain the visual fusion image and the semantic fusion image; The loss function used by the visual fusion branch during the training process is used to make the output image of the visual fusion branch highlight the salient features of the input image in the thermal radiation region, and drive the fusion result to be smoother in the non-thermal radiation region.

2. The robust image fusion method based on visual semantic collaboration according to claim 1, characterized in that, The encoding module consists of multiple sequentially connected encoding network layers, each encoding network layer including a cascaded interactive gated hybrid attention module and a progressively coupled attention module; The interactive gated hybrid attention module is used to perform local cross-modal interaction of visible light features and infrared features. After correcting misjudgments in pooling attention through a gating mechanism, the visible light features and infrared features are updated. The visible light features and infrared features of the first-level coding network layer are extracted from the input visible light image and infrared image, while the visible light features and infrared features of the remaining coding network layers are the outputs of the interactive gated hybrid attention module in the previous coding network layer. The progressively coupled attention module performs cross-modal global integration of the updated visible light and infrared features and outputs the depth features corresponding to the coding network layer. The output of the encoding module is composed of the deep features output by the progressively coupled attention modules of each level of the encoding network layer.

3. The robust image fusion method based on visual semantic collaboration according to claim 1, characterized in that, The steps of integrating the visual fusion features and the semantic fusion features in the feature search module of the multi-task learning model to generate a shared feature pool, and obtaining enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool, specifically include: The visual fusion features and the semantic fusion features are integrated to generate a shared feature pool; After the visual fusion feature is concatenated with the shared feature pool, it is processed by a 1-0 convolution to generate an enhanced visual fusion feature; The semantic fusion feature is concatenated with the shared feature pool and then processed by a 1-0 convolution to generate an enhanced semantic fusion feature.

4. The robust image fusion method based on visual semantic collaboration according to any one of claims 1-3, characterized in that, The loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the input visible light image and infrared image. In non-thermal radiation regions, the fusion result driven by the visual fusion branch approaches the weighted average of the pixel intensities at corresponding positions in the input visible light image and infrared image.

5. The robust image fusion method based on visual semantic collaboration according to any one of claims 1-3, characterized in that, The loss function is configured as follows: Based on semantic fusion labels, the image output by the visual fusion branch is divided into thermal radiation regions and non-thermal radiation regions. In the thermal radiation region, the fusion result that drives the visual fusion branch output approaches the maximum pixel intensity at the corresponding position in the visible light optimized image and the infrared optimized image. In non-thermal radiation regions, the fusion result driving the visual fusion branch output approaches the weighted average of the pixel intensities at corresponding positions in the visible light optimized image and the infrared optimized image. The visible light optimized image and the infrared optimized image are clean images obtained by image enhancement processing of the input visible light image and infrared image, respectively.

6. The robust image fusion method based on visual semantic collaboration according to any one of claims 1-3, characterized in that, The multi-task learning model uses a gradient correction strategy during training, which includes gradient conflict correction and gradient balancing aggregation. The gradient conflict correction is used to randomly select the gradient of one task and project it onto the normal plane of the gradient of the other task when the angle between the gradients of the visual fusion task and the semantic fusion task is an obtuse angle. The gradient balancing aggregation is used to aggregate weighted gradients into shared gradients for parameter updates in multi-task learning models.

7. A robust image fusion system based on visual semantic collaboration, characterized in that, A robust image fusion method based on visual semantic collaboration as described in any one of claims 1-6, comprising: The acquisition module is used to acquire visible light images and infrared images; The dual-branch extraction module is used to extract visual fusion features and semantic fusion features through the encoding modules of the visual fusion branch and the semantic fusion branch, respectively. The feature search module is used to integrate the visual fusion features and the semantic fusion features to generate a shared feature pool, and to obtain enhanced visual fusion features and enhanced semantic fusion features based on the shared feature pool; The dual-branch output module is used to input the enhanced visual fusion features and the enhanced semantic fusion features into the decoding modules corresponding to the visual fusion branch and the semantic fusion branch, respectively, to obtain the visual fusion image and the semantic fusion image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robust image fusion method based on visual semantic collaboration as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robust image fusion method based on visual semantic collaboration as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robust image fusion method based on visual semantic collaboration as described in any one of claims 1 to 6.

Images