Composite degraded image fusion method, system and device and storage medium

Abstract

The invention provides a composite degraded image fusion method, system and device and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: mapping a multi-modal image to a potential space through a pre-trained automatic encoder; performing information recovery in the potential space by adopting a potential diffusion framework based on a mean regression stochastic differential equation to obtain clean potential features of the multi-modal image after degeneration is removed; and based on the clean potential features, performing feature fusion through a potential diffusion fusion module of a U-net architecture, and according to a feature fusion result, decoding to obtain a fused image. According to the method, the applicability of image fusion in an extreme composite degradation scene is effectively improved, on the basis, preference modulation of a specific target object in the image can be achieved according to a language instruction input by a user, and various visual and semantic requirements of the user are flexibly met.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, device, and storage medium for fusing composite degraded images. Background Technology

[0002] In real-world imaging environments, due to the complexity of imaging equipment, lighting conditions, and atmospheric interference, single-modal images often fail to fully and accurately represent scene information, thus limiting the comprehensive perception capabilities of humans or machines and posing potential risks to subsequent decision-making. Against this backdrop, multimodal image fusion technology has emerged. This technology aims to integrate complementary information from multiple source images to generate a fused image that is richer in information and more expressive. With its superior performance in enhancing visual perception, image fusion has become an indispensable key technology in vision systems and is widely used in many downstream scenarios such as assisted driving, intelligent security, and military surveillance.

[0003] However, existing fusion methods still face many challenges that need to be addressed. Their performance limitations potentially restrict the reliability of multimodal image fusion technologies, such as infrared and visible light images, in real-world scenarios and introduce security risks.

[0004] First, due to hardware limitations and the influence of complex imaging environments, the acquired multimodal source images often suffer from significant quality degradation. For example, visible light images are typically affected by insufficient illumination, noise interference, and color shift, while infrared images generally suffer from low contrast, high noise, and uneven stripes. These degradation factors, especially when they occur in combination, can severely interfere with the evaluation of the effectiveness of image information, thereby undermining the rationality of information complementarity during the fusion process.

[0005] Secondly, due to the lack of objective, real-world fusion results as a standard, image fusion is essentially a no-reference perception problem. This means that different human observers or downstream machine vision algorithms often have different needs for the fused image. However, current image fusion methods typically seek a single, general fusion function within a closed framework, and the fusion strategy becomes fixed after optimization. This closed fusion framework lacks flexibility and cannot meet the personalized visual and semantic perception needs of different users. Summary of the Invention

[0006] This invention provides a method, system, device, and storage medium for fusing composite degraded images, addressing the shortcomings of existing image fusion methods in terms of applicability to composite degraded scenarios and improving the robustness of image fusion in such scenarios. Furthermore, this invention proposes a language-driven composite degraded image fusion method to meet the visual and semantic perception needs of different users.

[0007] In a first aspect, the present invention provides a method for fusing composite degraded images, comprising: Multimodal images are mapped to a latent space using a pre-trained autoencoder; A latent diffusion framework based on mean regression stochastic differential equations is used to recover information in the latent space, thereby obtaining clean latent features of the multimodal image after removing degradation. Based on the clean latent features, feature fusion is performed through the latent diffusion fusion module of the U-net architecture, and the fused image is decoded according to the result of the feature fusion.

[0008] According to a composite degraded image fusion method provided by the present invention, the pre-trained autoencoder uses a same-modal recombination loss function between images belonging to the same modality but with different degradations, and uses a cross-modal recombination loss function between images of different modalities but with different degradations in the same scene; wherein, the same-modal recombination loss function is used to constrain the spatial mapping error between images with different degradations in the same modality; the cross-modal recombination loss function is used to constrain the cross-modal invariance of the multi-layer intermediate features obtained by mapping multimodal images to the latent space.

[0009] According to the present invention, a composite degraded image fusion method is provided, wherein a latent diffusion framework based on mean regression stochastic differential equations is used to recover information in the latent space to obtain clean latent features of the multimodal image after degrade removal, including: Obtain the latent features obtained by mapping the multimodal image to the latent space; Design a forward diffusion process based on mean regression, and add noise to the latent features through the forward diffusion process to obtain noisy latent features; A backdiffusion process based on stochastic differential equations is designed to perform reverse reasoning on the noise latent features, thereby obtaining the clean latent features of the multimodal image after removing degradation.

[0010] According to a composite degraded image fusion method provided by the present invention, before feature fusion based on the clean latent features using a latent diffusion fusion module of the U-net architecture, the method further includes: Obtain language commands input by the user, and extract multi-scale location prior information based on the language commands; The process of performing feature fusion based on the clean latent features using a latent diffusion fusion module of the U-net architecture, and decoding the fused image based on the result of the feature fusion, includes: The clean latent features and the multi-scale location prior information are input into the latent diffusion fusion module of the U-net architecture for feature fusion to obtain fused latent features; The multi-layer intermediate features obtained by mapping the multimodal image to the latent space are fused using an averaging rule to obtain fused intermediate features; The final fused image is obtained by decoding the fusion potential features and the fusion intermediate features through the decoder of the autoencoder.

[0011] According to a composite degradation image fusion method provided by the present invention, before inputting the clean latent features and the multi-scale location prior information into the latent diffusion fusion module of the U-net architecture for feature fusion to obtain the fused latent features, the method further includes: Based on the target region corresponding to the language instruction, a visual enhancement regularization loss function is constructed. Based on the semantic features of the target region corresponding to the language instruction, a semantic guidance loss function is established based on weighted cross-entropy. The comprehensive loss function of the potential diffusion fusion module is constructed based on the visual enhancement regularization loss function and the semantic guidance loss function.

[0012] According to the present invention, a composite degraded image fusion method is provided, wherein establishing a semantically guided loss function based on weighted cross-entropy according to the semantic features of the target region corresponding to the language instruction includes: The final fused image obtained by the decoder is semantically segmented using a segmentation network; The weight of the target region corresponding to the language instruction is determined based on the result of the semantic segmentation. Based on the weights of the target region, a semantic guidance loss function is established in conjunction with the cross-entropy loss function.

[0013] According to the present invention, a composite degraded image fusion method is provided, the method further comprising: Based on the semantic segmentation results, a semantic segmentation loss function for the segmentation network is established using weighted cross-entropy. The segmentation network and the potential diffusion fusion module are trained collaboratively using the semantic segmentation loss of the segmentation network and the comprehensive loss function of the potential diffusion fusion module.

[0014] Secondly, the present invention also provides a composite degraded image fusion system, comprising: The spatial mapping module is used to map multimodal images to a latent space using a pre-trained autoencoder; The information recovery module is used to perform information recovery in the latent space using a latent diffusion framework based on mean regression stochastic differential equations to obtain clean latent features of the multimodal image after removing degradation. The instruction extraction module is used to acquire language instructions input by the user and extract multi-scale positional prior information based on the language instructions. The image fusion module is used to perform feature fusion based on the clean latent features and the multi-scale location prior information, through the latent diffusion fusion module of the U-net architecture, and decode the fused image according to the result of the feature fusion.

[0015] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the composite degradation image fusion method as described above.

[0016] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the composite degraded image fusion method as described above.

[0017] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the composite degraded image fusion method as described above.

[0018] The beneficial effects of the technical solutions provided by some embodiments of the present invention include at least the following: 1) The present invention provides a composite degraded image fusion method, which uses a latent diffusion framework based on mean regression stochastic differential equations to perform information recovery in the latent space, obtains clean latent features of the multimodal image after degradation removal, and uses them for feature fusion. This method can seamlessly combine information fusion with latent diffusion, effectively improve the applicability of image fusion in extreme composite degradation scenarios, and avoid the problem of information attenuation.

[0019] 2) This invention constrains single-modal images and cross-modal images by using the same-modal recombination loss function and the cross-modal recombination loss function, respectively, thereby establishing a strong correlation between degradation attributes and latent features, enhancing the expression of degradation attributes in latent features, and constraining the cross-modal invariance of multi-layer intermediate features.

[0020] 3) This invention designs a forward diffusion process based on mean regression and a backward diffusion process based on stochastic differential equations, which effectively estimates noise in the latent space, achieves complete backward inference, and embeds the prior information recovery into the latent diffusion process, which can improve the accuracy of multimodal image information recovery.

[0021] 4) Based on the aforementioned composite degraded image fusion method, this invention proposes a language-driven composite degraded image fusion method. This method can inject fine-grained positional prior information into the latent diffusion fusion module according to the user's input language commands, thereby dynamically aggregating multimodal features. This can achieve preference modulation of specific target objects in the image, realize interactive control of image fusion effect through language commands, and flexibly meet various visual and semantic needs of users.

[0022] 5) When setting the loss function, this invention achieves visual enhancement of specific target objects through visual enhancement regularization design, and guides the modulation of fused images in reverse through semantic segmentation, thereby enhancing the matching degree between semantic modulation and user language commands and improving the accuracy of language-driven image fusion.

[0023] 6) By establishing a collaborative optimization mechanism between feature fusion and semantic segmentation, this invention updates the segmentation network to adapt to the continuously updated fused images, which can promote the continuous enhancement of semantic guidance and ultimately achieve visual enhancement that mutually reinforces composite degradation robustness and high-quality visual fusion. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating a composite degraded image fusion method provided by the present invention; Figure 2 This is a schematic diagram illustrating the principle of composite degraded image fusion based on the latent diffusion fusion paradigm provided by the present invention. Figure 3 This is a schematic diagram of the potential diffusion fusion module of the U-net architecture provided by the present invention; Figure 4 This is a flowchart illustrating the language-driven composite degraded image fusion method provided by the present invention. Figure 5 This is a schematic diagram illustrating the principle of the language-driven controllable fusion strategy provided by the present invention. Figure 6 This is a schematic diagram of the collaborative optimization training of the segmentation network and the potential diffusion fusion module provided by the present invention; Figure 7 This is a schematic diagram showing the experimental comparison results of the fusion of the method of this invention and several typical methods in a complex degradation scenario provided by this invention; Figure 8This is a comparison diagram of the fusion effects of the language-driven composite degraded image fusion method and the language-free image fusion method provided by this invention; Figure 9 This is a schematic diagram of the structure of a composite degraded image fusion system provided by the present invention; Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] Existing multimodal composite degradation image fusion methods have the following main shortcomings: On the one hand, existing methods often focus on removing a single type of degradation, resulting in insufficient applicability in complex real-world scenes with multiple types of degradation, which weakens the texture and contrast of the fused image.

[0028] On the other hand, existing fusion models are insufficient in addressing the dynamic needs of practical applications, making it difficult to meet the visual and semantic preferences of different users and limiting the flexibility of actual deployment.

[0029] To address the above problems, this invention proposes a composite degradation image fusion method. This method effectively improves the robustness of image fusion in extreme composite degradation scenarios by seamlessly combining information fusion with potential diffusion. Furthermore, this invention proposes a language-driven controllable fusion strategy, allowing users to flexibly control the fusion process through language commands. Through both visual and semantic guarantees, it successfully achieves fine-grained semantic perception capabilities in the multimodal image fusion process.

[0030] Example 1 Please see Figure 1 , Figure 1 A flowchart illustrating a composite degraded image fusion method provided as an embodiment of the present invention is shown below. The method includes: S101. Map multimodal images to the latent space using a pre-trained autoencoder; S102. Using a latent diffusion framework based on mean regression stochastic differential equations, information recovery is performed in the latent space to obtain clean latent features after degrading multimodal images. S103. Based on clean latent features, feature fusion is performed through the latent diffusion fusion module of the U-net architecture, and the fused image is decoded according to the result of feature fusion.

[0031] This invention employs a latent diffusion framework based on mean regression stochastic differential equations to recover information in the latent space, obtaining clean latent features after degrading multimodal images, which are then used for feature fusion. This seamlessly combines information fusion with latent diffusion, effectively improving the applicability of image fusion in extreme complex degradation scenarios and avoiding information attenuation problems.

[0032] This invention, based on a potential diffusion framework and a potential diffusion fusion module, forms a potential diffusion-based fusion paradigm, such as... Figure 2 The diagram illustrates the principle of composite degraded image fusion based on a latent diffusion fusion paradigm provided by an embodiment of the present invention. The following section discusses... Figure 1 , Figure 2 The specific implementation process of this embodiment will be described below.

[0033] In S101 of this embodiment, the pre-trained autoencoder includes an encoder E and a decoder D. The encoder maps the input multimodal image to a latent space, obtaining the latent features and multi-layer intermediate features corresponding to each multimodal image, expressed as:

[0034] Where I represents the original input multimodal image. Indicates the modal type, such as visible light image, infrared image; Indicates encoder, Indicates latent features, This represents multiple layers of intermediate features output through skip connections. This indicates compound degradation, i.e. This represents a composite degenerate image of modality type M from the original input. This represents a composite degenerate latent feature of modal type M in the latent space. This represents a composite degenerate multilayer intermediate feature of modal type M in the latent space.

[0035] The decoder receives the latent features and multi-layer intermediate features encoded by the encoder, and maps them back from the latent space to the image space. Its expression is:

[0036] in, Indicates decoder, This represents a composite degenerate image of modality type M in the image space obtained by the decoder.

[0037] like Figure 2 As shown, the autoencoder in this embodiment can have multiple encoders, each encoder converting the composite degraded visible light image into a single encoder. Composite degraded infrared images Encoding into the latent space yields a composite degraded visible light image. Corresponding latent features and multi-layer intermediate features and composite degraded infrared images Corresponding latent features and multi-layer intermediate features Furthermore, different encoders can share weights.

[0038] This invention proposes a cross-modal feature recombination mechanism to guide the optimization of autoencoders, so that degenerate attributes should be concentrated in latent features as much as possible, and multi-layer intermediate features should focus on supplementing scene information, and have cross-modal invariance.

[0039] In some possible implementations, the cross-modal feature recombination mechanism includes: For images belonging to the same modality but with different degradation, the same modality recombination loss function is used to constrain the correlation between the degradation attributes and latent features of each modality image; A cross-modal recombination loss function is used to constrain the cross-modal invariance of multi-modal images with different modalities and different degradations in the same scene.

[0040] Specifically, the same-modal recombination loss function It can be represented as:

[0041] in, Indicates the modal type index. and Indexes representing degenerate types, when The condition indicates no degradation; this constraint establishes a strong correlation between degradation attributes and latent features, and should concentrate degradation attributes as much as possible in latent features.

[0042] Cross-modal recombination loss function It can be represented as:

[0043] in, and Indicates a degenerate type index. and This represents the modality type index. For cross-modal images of the same scene, this constraint ensures that the reconstruction result is independent of the source modality of the multi-layer intermediate features.

[0044] This invention constrains single-modal images and cross-modal images by using intramodal recombination loss functions and cross-modal recombination loss functions, respectively, thereby establishing a strong correlation between degradation attributes and latent features, enhancing the expression of degradation attributes in latent features, and constraining the cross-modal invariance of multi-layer intermediate features.

[0045] In S102 of this embodiment, a latent diffusion framework based on mean regression stochastic differential equations is adopted to achieve diffusion optimization in the latent space and drive the embedding of prior information recovery.

[0046] In some possible implementations, a latent diffusion framework based on mean regression stochastic differential equations is employed to recover information in the latent space, obtaining clean latent features after degrading multimodal images, including: Obtain latent features by mapping multimodal images to a latent space; Design a forward diffusion process based on mean regression, and add noise to the latent features through the forward diffusion process to obtain noisy latent features; We design a backdiffusion process based on stochastic differential equations to infer the noise latent features in reverse, thereby obtaining clean latent features after degrading the multimodal image.

[0047] The forward diffusion process of this potential diffusion framework can be formalized as follows:

[0048] in, Indicates the time step. This represents Gaussian noise. This represents the parameter that controls the speed of mean regression. The dispersion factor represents the degree of noise addition controlled. It is the mean, ensuring that after a certain number of time steps, the forward process will converge to a mean of . It has a stable Gaussian distribution.

[0049] Corresponding to the forward process, the reverse diffusion process of this potential diffusion framework can be formalized as follows:

[0050] in, This represents the latent characteristics at time t in the potential diffusion process. The probability distribution; This represents the score function of a stochastic differential equation, which is a latent feature. log density gradient, Indicates the latent features Find the gradient. for The estimated value.

[0051] This invention utilizes paired composite degradation feature maps and non-degradation feature maps. Train a denoising network to estimate the noise in the latent space. According to noise The transformation relationship between the score function and the score function The design and optimization of the potential diffusion process are carried out to achieve complete reverse inference, thereby embedding the information recovery prior into the potential diffusion process.

[0052] like Figure 2 As shown, for composite degraded visible light images Potential characteristics Composite degraded infrared images Potential characteristics Through the complete reverse reasoning process of this potential diffusion framework, the results can be recovered separately. Potential characteristics Clean latent features after removal of degradation ,as well as Potential characteristics Clean latent features after removal of degradation , .

[0053] This invention designs a forward diffusion process based on mean regression and a backward diffusion process based on stochastic differential equations, which effectively estimates noise in the latent space, achieves complete backward inference, and embeds prior information recovery into the latent diffusion process, thereby improving the accuracy of multimodal image information recovery.

[0054] In S103 of this embodiment, the Latent Diffusion Fusion Module (LFFM) of the U-net architecture aggregates the clean latent features obtained through the complete reverse reasoning process to achieve latent diffusion fusion, and finally decodes and outputs the fused image.

[0055] like Figure 3The diagram shows the structure of the latent diffusion fusion module LFFM in the U-net architecture provided in this embodiment. This latent diffusion fusion module includes multiple information processing modules, each comprising a residual block and a spatial transformer connected sequentially. After performing convolution operations on the input data, the latent diffusion fusion module achieves downsampling of the U-net architecture through alternating processing of the information processing modules and downsampling operations, and achieves upsampling of the U-net architecture through alternating processing of the information processing modules and upsampling operations. Downsampling and upsampling are connected via a skip connection.

[0056] Specifically, two weight matrices are generated through the latent diffusion fusion module. and Clean latent features of the input visible light image and clean latent features of infrared images Perform weighted integration and output the fused latent features obtained from the aggregation, expressed as:

[0057] in, The fusion potential features obtained from aggregation.

[0058] Then, the intermediate features from multiple layers are fused using an averaging rule:

[0059] in, These are composite degraded visible light images. Multi-layered intermediate features in latent space, composite degraded infrared images Multi-layered intermediate features in the latent space; These are the intermediate features obtained through fusion using the average rule.

[0060] Finally, the fused image is reconstructed using the decoder D of the autoencoder:

[0061] in, for Figure 2 The image shown is a fused image obtained using a fusion paradigm based on potential diffusion.

[0062] This invention proposes a fusion paradigm based on latent diffusion, which effectively improves the robustness of image fusion in extreme complex degradation scenarios by seamlessly combining the latent diffusion fusion module with the latent diffusion framework.

[0063] Example 2 Based on the aforementioned composite degraded image fusion method, this invention also proposes a language-driven composite degraded image fusion method, which can control the image fusion effect through language command interaction, flexibly adapting to the visual and semantic needs of different users.

[0064] Please see Figure 4 ,like Figure 4 The diagram shown is a flowchart of a language-driven composite degraded image fusion method provided by an embodiment of the present invention. The language-driven composite degraded image fusion method includes: S201. Map multimodal images to the latent space using a pre-trained autoencoder; S202. Using a latent diffusion framework based on mean regression stochastic differential equations, information recovery is performed in the latent space to obtain clean latent features after degrading multimodal images. S203. Obtain the user's input language commands and extract multi-scale location prior information based on the language commands; S204. Input the clean latent features and multi-scale location prior information into the latent diffusion fusion module of the U-net architecture to perform feature fusion and obtain fused latent features; S205. The multi-layer intermediate features obtained by mapping the multimodal image to the latent space through the averaging rule are fused to obtain the fused intermediate features; S206. The latent features and intermediate features of fusion are decoded by the decoder to obtain the final fused image.

[0065] This invention parses user-input language commands and injects fine-grained location prior information into the potential diffusion fusion module of the U-net architecture, constructing a language-driven controllable fusion strategy. This dynamically aggregates multimodal features and enables preference modulation of specific target objects in an image.

[0066] like Figure 5 The diagram shown is a schematic representation of the principle of a language-driven controllable fusion strategy provided in an embodiment of the present invention.

[0067] The following is combined Figure 4 , Figure 5 The specific implementation method of this embodiment will be described.

[0068] S201~S202 of this embodiment can be referred to S101~S102 of Embodiment 1, and will not be repeated here.

[0069] In S203 of this embodiment, the user's input language command is obtained, and the language command is parsed using the Grounded-SAM (Grounded Segment Anything Model) basic model. The input language command is converted into a mask in the image that refers to the target of interest, and the feature size is changed to obtain a series of multi-scale positional prior information. These multi-scale location priors will provide spatial positioning clues for controllable fusion strategies.

[0070] In S204 of this embodiment, the multi-scale position prior information obtained by parsing the language instructions in S203 is... Dense integration is achieved into each information processing module of the latent diffusion fusion module LFFM within the U-net architecture, as well as into each upsampling and downsampling layer, to ensure appropriate conditional coupling. The integration location of multi-scale location prior information is as follows: Figure 5 As shown, where, These represent prior location information at different scales.

[0071] Feature fusion is performed using the latent diffusion fusion module of the U-net architecture, resulting in the following expression for the fused latent features:

[0072] in, To integrate potential features.

[0073] Then, the intermediate features of multiple layers are fused using the averaging rule. .

[0074] Finally, based on the fusion potential features and the fusion of intermediate features The final merged image is obtained by reconstructing the merged image using a decoder.

[0075] in, This is the final fused image generated by the language-driven controllable fusion strategy of this invention.

[0076] Understandably, before performing feature fusion using the latent diffusion fusion module of the U-net architecture, the loss function of the latent diffusion fusion module should be designed, and the module should be trained.

[0077] In some possible implementations, before inputting clean latent features and multi-scale location prior information into the latent diffusion fusion module of the U-net architecture for feature fusion to obtain fused latent features in step S204, the following steps are also included: S204-1) Construct a visual enhancement regularization loss function based on the target region corresponding to the language instruction; S204-2) Based on the semantic features of the target region corresponding to the language instruction, a semantic guidance loss function is established based on weighted cross-entropy; S204-3) A comprehensive loss function for constructing a potential diffusion fusion module based on the visual enhancement regularization loss function and the semantic guidance loss function.

[0078] In S204-1 of this embodiment, through visual enhancement regularization design, the optimization target of enhancing the visual saliency of the target region of interest of the language instruction is clearly defined, and the target region is visually enhanced to improve the flexibility of image fusion.

[0079] Specifically, a frequency-based visual reference image is defined:

[0080] in, The fused image is obtained based on the S103 latent diffusion-based fusion paradigm. For a single image used to supervise the final fused image Optimized visual reference image, and These represent the high-frequency and low-frequency components, respectively. and This indicates the target region and other regions corresponding to the language instructions. It is a gain factor.

[0081] The visual enhancement regularization loss function can be formalized as:

[0082] By defining the above frequency-based visual reference image to construct the visual enhancement regularization loss function, the global consistency of low-frequency components can be used to provide a basic guarantee for the brightness and color attributes in the fused image, while the target regions are processed separately. Other regions The high-frequency components can achieve visual enhancement of contrast.

[0083] In S204-2 of this embodiment, the semantic quality of the specified object under language instructions is improved by constructing a semantic reverse guidance mechanism.

[0084] In some possible implementations, a semantic guidance loss function is established based on weighted cross-entropy, according to the semantic features of the target region corresponding to the language instruction, including: The final fused image obtained from the decoder is semantically segmented using a segmentation network; The weights of the target regions corresponding to language instructions are determined based on the results of semantic segmentation. Based on the weights of the target regions, a semantic guidance loss function is established in conjunction with the cross-entropy loss function.

[0085] Specifically, the final fused image generated by the language-driven controllable fusion strategy of this invention is processed through a segmentation network. Semantic segmentation is performed to improve the understanding of image scenes:

[0086] in, It is the classification probability predicted by the segmentation network. It is a frozen segmented network.

[0087] Then, the modulation of the fused image is guided in reverse by constraining the segmentation score, and a semantically guided loss function is established based on weighted cross-entropy. This makes semantic modulation conform to language instructions:

[0088] in, Let c represent the total number of semantic categories, where c = 1, 2, ..., C. This represents the segmentation label for the c-th semantic category predicted by the segmentation network. It is a category weight coefficient related to the semantic category corresponding to the language instruction.

[0089] By assigning higher weights to the semantic categories corresponding to language instructions, semantic attributes are guided to be injected into the target region corresponding to the language instructions. middle:

[0090] in, Indicates the gain factor. It is a matcher used to represent the current semantic category. With language instructions match, These are the initial category correction coefficients.

[0091] Finally, in S204-2 of this embodiment, based on the visual enhancement regularization loss function... Semantic guidance loss function Constructing the comprehensive loss function of the potential diffusion fusion module :

[0092] At this point, during a single optimization process of the potential diffusion fusion module, the segmentation network is frozen.

[0093] In designing the loss function, this invention achieves visual enhancement of specific target objects through visual enhancement regularization design, and guides the modulation of the fused image in reverse through semantic segmentation, thereby enhancing the matching degree between semantic modulation and user language commands and improving the accuracy of language-driven image fusion.

[0094] In some possible implementations, S204-3) is followed by: S204-4) Based on the semantic segmentation results, establish the semantic segmentation loss function of the segmentation network based on weighted cross-entropy; (S204-5) Collaborative optimization training of the segmentation network and the potential diffusion fusion module is performed by using the semantic segmentation loss of the segmentation network and the comprehensive loss function of the potential diffusion fusion module.

[0095] like Figure 6 The diagram shows a schematic of the collaborative optimization training of the segmentation network and the potential diffusion fusion module.

[0096] like Figure 6 As shown in S204-4 of this embodiment, a semantic segmentation loss function is established. To drive the optimization of the segmentation network:

[0097] in, This indicates that when the latent diffusion fusion module is fixed after a single optimization, during the process of waiting for the segmentation network to optimize, the final fused image is predicted based on the language-driven controllable fusion strategy of this invention, using the latent diffusion fusion module.

[0098] also, Figure 6 In this invention, the segmentation network and the potential diffusion fusion module generate the final fused image through the language-driven controllable fusion strategy of this invention. Data interaction is performed through the comprehensive loss function of the potential diffusion fusion module. Semantic segmentation loss function of segmentation network Joint iterative training was conducted to achieve collaborative optimization of fusion and segmentation.

[0099] After completing steps S204-1) to S204-5), perform feature fusion based on the potential diffusion fusion module in S204, and subsequent steps to obtain the final fused image.

[0100] This invention temporarily fixes the diffusion fusion module after a single optimization to optimize the segmentation network, thereby helping the segmentation network adapt to the continuously updated fused image. During the optimization of the diffusion fusion module, the segmentation network is frozen and combined with their respective loss functions to establish a collaborative optimization mechanism between feature fusion and semantic segmentation. This collaborative optimization mechanism can promote the continuous enhancement of semantic guidance and also help correct the imperfections in the location prior, ultimately achieving visual enhancement that mutually reinforces composite degradation robustness and high-quality visual fusion.

[0101] The proposed composite degraded image fusion method, compared to existing fusion methods, achieves the construction of a potential diffusion fusion paradigm, the design of a language-driven controllable fusion strategy, and visual enhancement through mutual reinforcement of composite degradation robustness and high-quality visual fusion. To objectively evaluate the performance of the proposed algorithm, typical degradation-robust fusion and controllable enhancement fusion were selected for experimental comparison. The comparison results are as follows: Figure 7 , 8 As shown.

[0102] Figure 7This diagram illustrates the experimental comparison between the method of this invention and several typical methods in a complex degradation scenario. The degraded visible light image and degraded infrared image are the original multimodal images. The images in other columns represent comparisons of the fusion results corresponding to different methods. The comparison shows that, compared to typical methods such as LRRNet (A novel representation learning guided fusion network for infrared and visible images), Dif-Fusion (Dif-fusion: Toward high color fidelity in infrared and visible image fusion with diffusion models), DDFM (DDFM: denoising diffusion model for multi-modality image fusion), IF-FILM (Image Fusion via Vision-Language Model), Text-IF (Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion), MRFS (Mrfs: Mutuallyreinforcing image fusion and segmentation), and DDBF (Dispel darkness for better fusion: A controllable visual enhancer based on cross-modal conditional adversarial... The method of this invention produces a fused image with stronger texture and contrast, and a better image enhancement effect on the target area, demonstrating significant advantages.

[0103] Figure 8 This image shows a comparison of the fusion effects of the language-driven composite degraded image fusion method of this invention and the image fusion method without language control. Furthermore, Figure 8 It also provides comparisons across dimensions such as residual maps, location priors, and segmentation labels. The residual map represents the difference between image fusion methods with and without linguistic control, characterizing the degree to which linguistic instructions alter the fusion result. The location prior is the target mask provided by the Grounded-SAM model based on the linguistic instructions. (Comparison) Figure 8The residual map and location prior in the image can reflect the difference in performance between the language-driven composite degenerate image fusion method and the non-language-controlled image fusion method. The language-driven composite degenerate image fusion method of this invention realizes semantic fine-grained perception and controllable fusion based on language instructions.

[0104] comprehensive Figure 7 , Figure 8 As can be seen, the controllable enhancement fusion results of the language-driven composite degraded image fusion method proposed in this invention far exceed the performance of existing methods, greatly improving the visibility of scenes in complex degraded environments, and possessing good flexibility, supporting interactive control of the enhancement fusion effect through language commands.

[0105] Please see Figure 9 , Figure 9 A schematic diagram of a composite degraded image fusion system provided for an embodiment of the present invention, the system comprising: The spatial mapping module 910 is used to map multimodal images to a latent space via a pre-trained autoencoder; The information recovery module 920 is used to perform information recovery in the latent space using a latent diffusion framework based on mean regression stochastic differential equations to obtain clean latent features after degrading multimodal images. The instruction extraction module 930 is used to acquire language instructions input by the user and extract multi-scale position prior information based on the language instructions. The image fusion module 940 is used to perform image fusion based on clean latent features and multi-scale location prior information to obtain a fused image.

[0106] The composite degraded image fusion system and the composite degraded image fusion method described above can be referred to in correspondence.

[0107] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute the composite degradation image fusion method provided in the above method embodiments.

[0108] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the composite degraded image fusion method provided in the above method embodiments.

[0110] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the composite degraded image fusion method provided in the above-described method embodiments.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing composite degraded images, characterized in that, include: Multimodal images are mapped to a latent space using a pre-trained autoencoder; A latent diffusion framework based on mean regression stochastic differential equations is used to recover information in the latent space, thereby obtaining clean latent features of the multimodal image after removing degradation. Based on the clean latent features, feature fusion is performed through the latent diffusion fusion module of the U-net architecture, and the fused image is decoded according to the result of the feature fusion.

2. The composite degraded image fusion method according to claim 1, characterized in that, The pre-trained autoencoder uses a same-modal recombination loss function between images belonging to the same modality but with different degradations, and a cross-modal recombination loss function between images of different modalities but with different degradations in the same scene. The same-modal recombination loss function is used to constrain the correlation between the degradation attributes and latent features of each modality image. The cross-modal recombination loss function is used to constrain the cross-modal invariance of the multi-layer intermediate features obtained by mapping multimodal images to the latent space.

3. The composite degraded image fusion method according to claim 1, characterized in that, The method employs a latent diffusion framework based on mean regression stochastic differential equations to recover information in the latent space, obtaining clean latent features of the multimodal image after degradation removal, including: Obtain the latent features obtained by mapping the multimodal image to the latent space; Design a forward diffusion process based on mean regression, and add noise to the latent features through the forward diffusion process to obtain noisy latent features; A backdiffusion process based on stochastic differential equations is designed to perform reverse reasoning on the noise latent features, thereby obtaining the clean latent features of the multimodal image after removing degradation.

4. The composite degraded image fusion method according to claim 1, characterized in that, Before performing feature fusion based on the clean latent features using the latent diffusion fusion module of the U-net architecture, the method further includes: Obtain language commands input by the user, and extract multi-scale location prior information based on the language commands; The process of performing feature fusion based on the clean latent features using a latent diffusion fusion module of the U-net architecture, and decoding the fused image based on the result of the feature fusion, includes: The clean latent features and the multi-scale location prior information are input into the latent diffusion fusion module of the U-net architecture for feature fusion to obtain fused latent features; The multi-layer intermediate features obtained by mapping the multimodal image to the latent space are fused using an averaging rule to obtain fused intermediate features; The final fused image is obtained by decoding the fusion potential features and the fusion intermediate features using the decoder of the autoencoder.

5. The composite degraded image fusion method according to claim 4, characterized in that, Before inputting the clean latent features and the multi-scale location prior information into the latent diffusion fusion module of the U-net architecture for feature fusion to obtain the fused latent features, the method further includes: Based on the target region corresponding to the language instruction, a visual enhancement regularization loss function is constructed. Based on the semantic features of the target region corresponding to the language instruction, a semantic guidance loss function is established based on weighted cross-entropy. The comprehensive loss function of the potential diffusion fusion module is constructed based on the visual enhancement regularization loss function and the semantic guidance loss function.

6. The composite degraded image fusion method according to claim 5, characterized in that, The step of establishing a semantic guidance loss function based on weighted cross-entropy according to the semantic features of the target region corresponding to the language instruction includes: The final fused image obtained by the decoder is semantically segmented using a segmentation network; The weight of the target region corresponding to the language instruction is determined based on the result of the semantic segmentation. Based on the weights of the target region, a semantic guidance loss function is established in conjunction with the cross-entropy loss function.

7. The composite degraded image fusion method according to claim 6, characterized in that, The method further includes: Based on the semantic segmentation results, a semantic segmentation loss function for the segmentation network is established using weighted cross-entropy. The segmentation network and the potential diffusion fusion module are trained collaboratively using the semantic segmentation loss function of the segmentation network and the comprehensive loss function of the potential diffusion fusion module.

8. A composite degraded image fusion system, characterized in that, include: The spatial mapping module is used to map multimodal images to a latent space using a pre-trained autoencoder; The information recovery module is used to perform information recovery in the latent space using a latent diffusion framework based on mean regression stochastic differential equations to obtain clean latent features of the multimodal image after removing degradation. The instruction extraction module is used to acquire language instructions input by the user and extract multi-scale positional prior information based on the language instructions. The image fusion module is used to perform feature fusion based on the clean latent features and the multi-scale location prior information, through the latent diffusion fusion module of the U-net architecture, and decode the fused image according to the result of the feature fusion.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the composite degraded image fusion method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the composite degraded image fusion method as described in any one of claims 1 to 7.

Images