Infrared and visible light image fusion model construction method based on quality-semantic cooperation, infrared and visible light image fusion method and related device

Abstract

This invention relates to the field of image fusion technology, specifically a method for constructing an infrared and visible light image fusion model based on quality-semantic coordination, an infrared and visible light image fusion method, and related apparatus. The method includes proportionally dividing the data into a training sample set and a test sample set; training the QSFusion network using the training sample set, ending training when the total loss function stabilizes, thus obtaining the infrared and visible light image fusion model; and testing the trained infrared and visible light image fusion model using the test sample set, outputting a vibration spectrum fusion classification model that meets the test evaluation requirements. This invention establishes the QSFusion network, an endogenous semantic-driven image fusion network under quality-semantic joint supervision. It deeply mines the semantic potential within the network, constructing a feature bridge connecting low-level vision and high-level cognition, achieving adaptive semantic perception while avoiding parameter redundancy in high-level visual task models.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and is a method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, an infrared and visible light image fusion method, and related devices. Background Technology

[0002] Visible light images and infrared images focus on different types of image information. Visible light images contain rich texture details, while infrared images contain thermal radiation information of objects, effectively highlighting various salient targets, but lacking detailed descriptions of the scene. Fusion of visible light and infrared images can integrate thermal radiation targets with high-frequency texture details, improving the performance of advanced visual perception tasks such as target detection. Therefore, visible light and infrared image fusion technology is increasingly widely used in fields such as military reconnaissance, security monitoring, medical diagnosis, and autonomous driving.

[0003] Existing methods for fusing visible light and infrared images include:

[0004] The first category, such as Figure 1 The traditional image fusion method shown in (a) is dedicated to decoupling and reconstructing the gradient, intensity or frequency features of the source image pair at the pixel level. This type of method has achieved significant results in improving image contrast and low-level visual fidelity. However, its model optimization still mainly focuses on shallow feature fitting and lacks preservation of global semantic context information to a certain extent, which limits the performance improvement of advanced vision tasks such as object detection.

[0005] For example:

[0006] (1) Image fusion method based on deep learning

[0007] With the rapid development of deep learning technology, the research focus in the field of image fusion has gradually shifted from traditional mathematical methods such as early multi-scale transformations and sparse representations to network architectures based on feature learning. Ma et al. introduced the Swinransformer to construct a fusion model with global perception capabilities, effectively overcoming the limitation of local vision in conventional convolution; Li et al. proposed CrossFuse, which uses a cross-attention mechanism to achieve finer feature alignment in spatial and channel dimensions; Xu et al. proposed the MURF framework, which significantly improves the structural fidelity of fused images by designing mutually reinforcing modal interaction modules; the EMMA framework proposed by Li et al. focuses on the effective alignment of multimodal features, alleviating the ghosting problem caused by non-strict registration of source images; Fan et al. proposed D2Fusion for feature decoupling in two domains, while Park et al. proposed CMTFusion to aggregate global contextual information.

[0008] The deep learning-based methods mentioned above have achieved significant results in improving image contrast and visual smoothness. However, these methods mainly rely on pure low-level pixel-level reconstruction and lack high-level semantic guidance. When dealing with complex scenes, they tend to neutralize thermal radiation edges and texture details, which to some extent limits the performance ceiling of downstream perception tasks.

[0009] (2) Fusion methods for advanced vision tasks such as object detection

[0010] The ultimate goal of image fusion is not only to enhance the human visual experience but also to serve downstream machine vision tasks. Compared to semantic segmentation, which pursues pixel-level fine-grained classification, object detection tasks rely more on the global contour and saliency energy distribution of the object, which aligns perfectly with the initial intention of infrared and visible light fusion to extract structural and thermal radiation features. Therefore, joint fusion methods for advanced vision tasks such as object detection are gradually emerging.

[0011] Liu et al. designed the TarDAL two-layer optimization strategy, which directly backpropagates the object detection loss to the fusion module, effectively improving the compatibility between the fused image and the detection network; Tang et al. utilized the cascaded feedback of the semantic segmentation network to help the fusion model retain key contextual information; Liu et al. proposed the DCEvo cross-dimensional evolutionary learning framework, which dynamically balances the loss weights of fusion quality and downstream detection tasks; Bai et al. proposed a task-driven paradigm that introduces learnable fusion loss, using a meta-learning mechanism to alleviate the domain disconnect between the pre-trained detector and the unaligned fused image; Liang et al. proposed the TarDeFusion framework, which uses a task-aware mechanism to achieve dynamic focusing on salient targets in degraded scenes; Liu et al. proposed SegMiF, which completes the deep mapping of fusion and high-level semantic features through a cascaded architecture; Ma et al. proposed S4Fusion, which perceives multimodal global spatial semantics with linear complexity.

[0012] The task-driven paradigm described above successfully introduced advanced cognition into the fusion evaluation system, greatly enhancing the practical value of fused images. However, the cross-modal interaction of most methods is still limited to the shallow cascade of the terminal loss function, and the semantic requirements have not been fully internalized into the feature encoding stage. When faced with extreme lighting or significant modal differences, there is still a risk of insufficient extraction of key target semantics.

[0013] The second category, such as Figure 1 (b) shows the fusion method guided by the high-level vision task model. This type of method effectively enriches the semantic expression and cognitive attributes of the fused image by directly calling the high-level vision task model.

[0014] For example, Wu et al. proposed the Every SAM Drop Counts framework, which uses the spatial structure boundary extracted by SAM as an explicit prior, effectively making up for the weak semantic perception capability of traditional fusion networks; Yi et al. proposed Text-IF, which achieves degenerate perception multimodal feature alignment by locking the CLIP text encoder and introducing text prompts; Wang et al. explored an instance-level controllable fusion method under weak supervision by using a text semantic guidance mechanism.

[0015] The above method leverages the open-set semantic prior of advanced visual task models to effectively broaden the semantic boundaries of fused images. While the introduction of external semantic priors significantly enhances the high-dimensional representation capability of fused images, the massive parameter scale of advanced visual task models inevitably leads to high computational redundancy, and forcibly aligning heterogeneous external features can easily cause local structural distortion. Summary of the Invention

[0016] The purpose of this invention is to stimulate the adaptive semantic potential within the network without introducing advanced visual task models, and to solve the problems of insufficient multimodal feature interaction and semantic extraction disconnect. It provides a method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, an infrared and visible light image fusion method, and related devices.

[0017] One of the technical solutions of this invention discloses a method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, comprising:

[0018] Several samples were obtained and divided into a training sample set and a test sample set according to the proportion. Each sample included a visible light image and an infrared image to be fused, as well as the corresponding fused image.

[0019] The QSFusion network was trained using a training sample set. Training ended when the total loss function stabilized, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulate it to obtain infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention-weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image.

[0020] The performance of the trained infrared and visible light image fusion model was tested and quantitatively evaluated using a test sample set, and infrared and visible light image fusion models that meet the evaluation criteria were selected.

[0021] The optimized, adaptive semantic guidance module generates a spatially aware mask, including:

[0022] The Sobel operator was used to extract the high-frequency edge response of the grayscale components of infrared and visible light images, respectively.

[0023] Two high-frequency edge responses are spatially semantically integrated through two independent hierarchical convolutional branches to obtain two output features;

[0024] After acquiring the two output features, the features are concatenated along the channel dimension, and a spatially aware mask is generated by fusing two steps of convolution and compressed convolution.

[0025] The optimized dual-stream coding unit includes an infrared encoder and a visible light encoder, both of which consist of four consecutive Transformer modules. Each Transformer module includes a multi-head depth transposed attention network and a convolutional feedforward network.

[0026] The optimized, deep interaction fusion unit includes an attention weighted value aggregation operation module, a multi-scale cross-dimensional interaction module, and a two-way cross-attention module;

[0027] Two cross-attention modules are used to perform preliminary alignment of dual-modal features for infrared high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features through a cross-attention mechanism.

[0028] The attention-weighted aggregation operation module is used to perform feature attention-weighted aggregation on the aligned features. Then, the aggregated features are adaptively weighted and optimized through the attention-weighted aggregation operation to achieve deep interaction and representation enhancement of multimodal features. Finally, the enhanced features are output through residual connections.

[0029] By utilizing a multi-scale, cross-dimensional interactive module, the enhanced features are deeply enhanced to obtain collaborative features.

[0030] Preferably, the decoding and refinement output unit includes a decoder, a refinement module, an output convolutional block, and a sigmoid activation function;

[0031] The decoder uses four consecutive Transformer modules to gradually recover the spatial resolution and obtain the reconstructed features;

[0032] The reconstructed features are processed using a refinement module to eliminate residual artifacts;

[0033] After eliminating residual artifacts, a fused image is generated by applying the output convolutional block and the Sigmoid activation function.

[0034] Preferably, the total loss function is as follows:

[0035]

[0036] in, To compensate for the loss of fidelity; This is the semantic consistency alignment loss; Loss guided by quality; This is the regularization loss; , , All of these are weighting coefficients for each type of loss.

[0037] The second technical solution of the present invention discloses a method for fusion of infrared and visible light images, comprising:

[0038] Acquire a visible light image and an infrared image to be fused;

[0039] A visible light image and an infrared image to be fused are input into an infrared and visible light image fusion model to obtain the corresponding fused image. The infrared and visible light image fusion model is constructed using a quality-semantic collaboration-based infrared and visible light image fusion model construction method.

[0040] The third technical solution of the present invention discloses a device for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, comprising:

[0041] The sample acquisition unit acquires several samples and divides them into a training sample set and a test sample set according to a ratio. Each sample includes a visible light image and an infrared image to be fused, as well as the corresponding fused image.

[0042] The training unit trains the QSFusion network using a training sample set. Training ends when the total loss function stabilizes, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulates it to obtain infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention-weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image.

[0043] The testing unit uses a test sample set to perform performance testing and quantitative evaluation on the trained infrared and visible light image fusion models, and selects infrared and visible light image fusion models that meet the evaluation criteria.

[0044] The fourth technical solution of the present invention discloses an infrared and visible light image fusion device, comprising:

[0045] The basic data acquisition unit acquires a visible light image and an infrared image to be fused.

[0046] The fusion unit inputs a visible light image and an infrared image to be fused into the infrared and visible light image fusion model to obtain the corresponding fused image.

[0047] The fifth technical solution of the present invention discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the steps in the method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration or the infrared and visible light image fusion method.

[0048] This invention establishes the QSFusion network, an endogenous semantic-driven image fusion network under joint quality-semantic supervision. It deeply mines the semantic potential within the network, constructing a feature bridge connecting low-level vision and high-level cognition. While avoiding parameter redundancy in high-level visual task models, it achieves adaptive semantic perception. Specifically:

[0049] An adaptive semantic guidance module is set in the input modulation unit. The spatially aware mask is generated through the adaptive semantic guidance module to accurately enhance the intrinsic semantic features of the target.

[0050] The Transformer module, which incorporates a multi-head depth transpose attention mechanism and a convolutional feedforward network, is set up in the dual-stream coding unit. This ensures that the network can efficiently model global correlations between channels while significantly enhancing the discriminative power of hierarchical features for local high-frequency details and edge structures in the image.

[0051] The deep interactive fusion unit is equipped with a multi-scale cross-dimensional interaction module and a two-way cross-attention module to establish a deep cross-modal interaction mechanism, realize the dynamic interaction and collaboration of multi-scale features, dynamically filter redundant interference from the source, effectively make up for the loss of deep semantics, and thus solve the problem of insufficient multi-modal feature interaction and semantic extraction without introducing advanced visual task models. It stimulates the adaptive semantic potential inside the network, effectively raises the upper limit of perception performance while avoiding parameter redundancy of advanced visual task models, and realizes high-dimensional semantics to guide the generation of underlying pixels.

[0052] Furthermore, the present invention provides a quality-semantic joint supervision mechanism, which couples pixel-level fidelity with the cosine distance of high-dimensional latent space features and uses endogenous semantics to guide the generation of low-level pixels, ensuring the consistency of the fused image at the visual reconstruction and high-level semantic levels. Attached Figure Description

[0053] Appendix Figure 1 This is a schematic diagram of the existing visible light and infrared image fusion paradigm in the prior art.

[0054] Appendix Figure 2 This is a schematic diagram of the implementation environment in one embodiment of the present invention.

[0055] Appendix Figure 3 This is a schematic diagram of the process for constructing an infrared and visible light image fusion model based on quality-semantic collaboration in one embodiment of the present invention.

[0056] Appendix Figure 4 This is a schematic diagram of the QSFusion network structure in one embodiment of the present invention.

[0057] Appendix Figure 5 This is a schematic diagram of the adaptive semantic guidance module structure in one embodiment of the present invention.

[0058] Appendix Figure 6 This is a schematic diagram of the Transformer module structure in one embodiment of the present invention.

[0059] Appendix Figure 7 This is a schematic diagram of the multi-scale cross-dimensional interaction module structure in one embodiment of the present invention.

[0060] Appendix Figure 8 This is a schematic diagram of subjective visual comparison of the TNO dataset in one embodiment of the present invention.

[0061] Appendix Figure 9 This is a schematic diagram of subjective visual comparison of the M3FD dataset in one embodiment of the present invention.

[0062] Appendix Figure 10 This is a schematic diagram of subjective visual comparison of the RoadScene dataset in one embodiment of the present invention.

[0063] Appendix Figure 11 This is a schematic diagram of subjective visual comparison of the LLVIP dataset in one embodiment of the present invention.

[0064] Appendix Figure 12 This is a visual comparison diagram of target detection on the M3FD dataset in one embodiment of the present invention.

[0065] Appendix Figure 13 This is a schematic diagram of the infrared and visible light image fusion method in one embodiment of the present invention.

[0066] Appendix Figure 14 This is a schematic diagram of the structure of an infrared and visible light image fusion model construction device based on quality-semantic collaboration in one embodiment of the present invention.

[0067] Appendix Figure 15This is a schematic diagram of the infrared and visible light image fusion device in one embodiment of the present invention. Detailed Implementation

[0068] Those skilled in the art will understand that, unless specifically stated otherwise, in the embodiments of the present invention, a "module" or "unit" refers to a computer program or part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0069] In addition, in the embodiments of the present invention, "multiple" refers to two or more, and "first" and "second" are used to distinguish descriptions and should not be construed as implying relative importance.

[0070] The present invention will be further described below with reference to embodiments and accompanying drawings:

[0071] As attached Figure 1 , 2 As shown, the method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration includes: acquiring several samples and dividing them into a training sample set and a test sample set according to a certain ratio, wherein each sample includes a visible light image and an infrared image to be fused, as well as the corresponding fused image; training the QSFusion network using the training sample set, ending the training when the value of the total loss function is stable, and obtaining the infrared and visible light image fusion model; testing the trained infrared and visible light image fusion model using the test sample set, optimizing the model parameters, and outputting a vibration spectrum fusion classification model that meets the test evaluation requirements.

[0072] The method provided in this embodiment of the invention may involve artificial intelligence (AI) technology and may be implemented based on artificial intelligence technology, such as using deep learning to train a corresponding model using samples.

[0073] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence.

[0074] Deep learning (DL) specifically refers to machine learning based on deep neural network models and methods. It has developed from statistical machine learning, artificial neural network algorithms, and other algorithms, combined with the development of modern big data and high computing power. The most important technical feature of deep learning is its ability to automatically extract features.

[0075] As attached Figure 1 The diagram illustrates an implementation environment provided by an embodiment of the present invention. This implementation environment may include: training equipment and usage equipment.

[0076] Both the training equipment and the equipment used are computer devices; optionally, the computer device is a terminal device, such as a mobile phone, tablet computer, PC (Personal Computer) or other electronic devices; or, the computer device is a server, which can be a single server, a server cluster composed of multiple servers, or a cloud computing service center. This embodiment of the invention does not limit this.

[0077] Training equipment refers to computer equipment capable of training and learning QSFusion networks. Optionally, the training equipment has the ability to acquire QSFusion networks and train and learn them according to application requirements. For example, the training equipment acquires a QSFusion network from other devices via a network and then trains it using training samples according to application requirements, so that the QSFusion network has the ability to fuse infrared and visible light images. Optionally, the training equipment has the ability to construct QSFusion networks. It can construct QSFusion networks itself according to application requirements and then train and learn them. For example, in order to achieve infrared and visible light image fusion, the training equipment constructs a QSFusion network itself and then trains and learns it using samples according to application requirements.

[0078] The device being used refers to a computer device that has the requirement to use a QSFusion network. Optionally, the device being used obtains a QSFusion network from other devices through the network according to application requirements. For example, if the device being used has the requirement to fuse infrared and visible light images, it can obtain a QSFusion network that has been trained and learned for infrared and visible light image fusion from other devices through the network, and use the QSFusion network to perform infrared and visible light image fusion.

[0079] Based on this, the following will further illustrate the above-mentioned method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration with several embodiments.

[0080] Example 1: As shown in the attached document Figure 3As shown, this embodiment discloses a method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, including:

[0081] Step S110: Obtain several samples and divide them into a training sample set and a test sample set according to the proportion. Each sample includes a visible light image and an infrared image to be fused, as well as the corresponding fused image.

[0082] Step S120: Train the QSFusion network using the training sample set. Training ends when the total loss function stabilizes, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulates infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image.

[0083] Step S130: Use the test sample set to perform performance testing and quantitative evaluation on the trained infrared and visible light image fusion model, and select infrared and visible light image fusion models that meet the evaluation index requirements.

[0084] This invention discloses a method for constructing an infrared and visible image fusion model based on quality-semantic collaboration. It establishes a QSFusion network (Quality-Semantic Collaborative Network for Infrared and Visible Image Fusion), an endogenous semantic-driven image fusion network under joint quality-semantic supervision. In the input modulation unit, an adaptive semantic guidance module generates a spatially perceptual mask to accurately enhance the endogenous semantic features of the target. A deep cross-modal interaction mechanism is established in the dual-stream coding unit and the deep interactive fusion unit to achieve dynamic interaction and collaboration of multi-scale features. This dynamically filters redundant interference from the source, effectively compensating for the loss of deep semantics. Thus, without introducing advanced visual task models, it solves the problems of insufficient multimodal feature interaction and the disconnect between semantic extraction and the network's adaptive semantic potential. While avoiding computational redundancy, it effectively raises the upper limit of perceptual performance, enabling high-dimensional semantics to guide the generation of underlying pixels.

[0085] It should also be noted that the test evaluation metrics should be selected as needed during model testing.

[0086] Example 2: As shown in the attached document Figure 4 As shown, the QSFusion network in this embodiment includes:

[0087] The input modulation unit, located at the front end of the network, includes an adaptive semantic guidance module. This module generates a spatially aware mask, which is then used to perform complementary feature modulation on the input infrared and visible light images, respectively, to obtain infrared image enhancement features and visible light image enhancement features, as shown below:

[0088]

[0089] Where M is the spatial awareness mask; For mask generation function; The background compensation coefficient can be set to 0.05 in this embodiment; These are infrared images and visible light images, respectively. , These are infrared image enhancement features and visible light image enhancement features, respectively.

[0090] Based on the above description, the input modulation unit, through a background compensation mechanism, enhances the endogenous semantic expression of salient targets while ensuring the complete transmission of background cues.

[0091] Furthermore, the thermal radiation target in the infrared image and the background texture in the visible light image have physical differences in gradient distribution. In order to achieve local information enhancement before feature encoding, the Adaptive Semantic Guidance Module (ASG) in this embodiment is designed as a dual-branch convolutional generation module based on Sobel edge enhancement, and its structure is shown in the attached figure. Figure 5 As shown, the corresponding spatial awareness mask generation process includes:

[0092] First, to avoid multi-channel color interference, the Sobel operator is used to extract the high-frequency edge responses of the grayscale components of the infrared and visible light images respectively. ;

[0093]

[0094] in, These are Sobel convolution kernels in the horizontal and vertical directions, respectively; edge responses are superimposed on the original input as spatial priors to form an enhanced signal.

[0095] Subsequently, two high-frequency edge responses Spatial semantic integration is performed through two independent hierarchical convolutional branches to obtain two output features;

[0096]

[0097] Next, after obtaining the two output features, the concatenation is performed in the channel dimension, and a spatially aware mask M is generated by fusing convolution and compressed convolution in two logical steps.

[0098]

[0099] in, This is the Sigmoid activation function.

[0100] The dual-stream coding unit includes an infrared encoder and a visible light encoder. Both the infrared encoder and the visible light encoder are composed of four consecutive Transformer modules. While extracting features, they perform downsampling step by step to extract hierarchical features at different spatial scales in the form of sets. This captures cross-scale high-dimensional endogenous semantic features in infrared image enhancement features and visible light image enhancement features, namely infrared high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features.

[0101]

[0102] in, For feature level; , These are external high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features, respectively.

[0103] The dual-stream encoder setup of the infrared encoder and the visible light encoder described above captures high-dimensional endogenous semantic information across scales through internal linear complexity units, and extracts deep endogenous semantics with global correlation.

[0104] Furthermore, to improve the computational efficiency of hierarchical encoding, in this embodiment, both the infrared encoder and the visible light encoder consist of four consecutive Transformer modules, as shown in the attached figure. Figure 6 As shown, the original architecture utilizes the Multi-Head Deep Transposed Attention (MDTA) mechanism to establish global associations in the channel dimension, effectively avoiding the quadratic computational overhead of traditional self-attention mechanisms in the spatial dimension. However, considering the requirements of image fusion tasks for refined local texture details, the gated deep feedforward network (GDFN) in the original architecture, which includes a nonlinear gating mechanism, is prone to feature mapping redundancy when processing multimodal heterogeneous features. Therefore, this embodiment replaces the complex gating structure with a convolutional feedforward network (CFFN). In specific implementation, by concatenating point convolutions and standard spatial convolutions, the local operation characteristics of the convolution operator are utilized to inject a key spatial inductive bias into the features. This ensures that while the network efficiently models global associations between channels, it can significantly enhance the discriminative power of hierarchical features for local high-frequency details and edge structures in the image.

[0105] The deep interaction fusion unit includes an attention weighting aggregation module, a multi-scale cross-dimensional interaction module, and a two-way cross-attention module, specifically:

[0106] Two cross-attention modules are used to perform preliminary alignment of dual-modal features of infrared high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features through the cross-attention (CA) mechanism.

[0107] The attention weighted value aggregation operation module is used to perform feature attention weighted value aggregation on the aligned features. Then, the aggregated features are adaptively weighted and optimized through the attention weighted value aggregation operation (AWVA) to achieve deep interaction and representation enhancement of multimodal features. Finally, the enhanced features are output through residual connections.

[0108]

[0109] in, This is a channel-dimensional concatenation operation, and the channel number alignment is completed by a 1×1 convolution after concatenation; F is the aggregated feature. To enhance features;

[0110] The enhanced features are deeply augmented using a multi-scale cross-dimensional interaction module (MCI) to obtain collaborative features. .

[0111]

[0112] The aforementioned deep interaction fusion unit utilizes a cross-modal cross-gating strategy to achieve dynamic interaction and collaboration of multi-scale features, effectively compensating for the loss of deep semantics.

[0113] Furthermore, the Attention Weighted Value Aggregation (AWVA) operation adaptively weights the normalized cross-modal feature values ​​through a multi-head attention mechanism, enhancing the complementary information between external high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features, suppressing redundant noise between modalities, maintaining linear complexity in the computation process, and not introducing additional module-level parameter redundancy.

[0114] Furthermore, to compensate for the structural details lost during downsampling in deep networks, this embodiment designs a multi-scale cross-dimensional interaction module (MCI), the structure of which is shown in the attached figure. Figure 7 As shown, the process of deep enhancement of enhanced features includes:

[0115] First, the input augmented features Mapping to a three-scale space, constructing multi-scale feature branches, and obtaining multi-scale features;

[0116]

[0117] Subsequently, features at each scale are fed into the corresponding cross-dimension enhancement block (CrossDimBlock) for feature activation. Within this block, features are processed in parallel through spatial convolution branches and channel attention branches to capture local details, and residual connections preserve the basic semantics. The enhanced features of each branch are then uniformly upsampled to the original resolution (where...). Branch road directly as ), execute channel cascading and Convolutional fusion yields enhanced features. ;

[0118]

[0119] Finally, learnable adaptive weighting coefficients (i.e., learnable residual gain) are introduced. Applying this to the main branch input generates the final collaborative features. :

[0120]

[0121] in, These are learnable adaptive weighting coefficients.

[0122] The aforementioned multi-scale, cross-dimensional interaction module ensures that the network can spontaneously balance the weights between high-level semantics and multi-scale supplementary details, thereby generating collaborative features with strong discriminative power.

[0123] The decoding and refinement of the output unit includes the decoder, refinement module, output convolutional block, and sigmoid activation function, specifically:

[0124] The decoder employs a hierarchical upsampling structure combined with the encoder's skip fusion, using four consecutive Transformer modules to gradually restore spatial resolution and obtain reconstructed features;

[0125] The reconstructed features are processed using the refinement module to eliminate residual artifacts;

[0126] A fused image is generated by applying the output convolutional block to the Sigmoid activation function;

[0127]

[0128] in, For the decoding process; This is the Sigmoid activation function.

[0129] The QSFusion network structure in this embodiment can deeply explore the semantic potential within the network, construct a feature bridge connecting the underlying vision and the high-level cognition. While avoiding the redundancy of model parameters in high-level vision tasks, it achieves adaptive semantic perception, thereby effectively increasing the upper limit of perception performance while avoiding computational redundancy, and realizing high-dimensional semantics to guide the generation of underlying pixels.

[0130] Example 3: In this example, to guide the network towards machine-aware optimization, a quality-semantic joint supervision framework is constructed to perform joint optimization of quality and semantics. This framework includes a Quality-Guided (QG) branch, which achieves comprehensive constraints from pixel to semantics through joint optimization of three loss functions. The total loss function is as follows:

[0131]

[0132] in, To compensate for the loss of fidelity; This is the semantic consistency alignment loss; Loss guided by quality; This is the regularization loss; , , All are weighting coefficients for each loss, where This is the L2 regularization term for weights, used to prevent network overfitting. In this embodiment, it can be set to... .

[0133] This quality-semantic joint supervision framework combines... (Pixel-level fidelity) and , (Perception and Task-Level Guidance) (Model constraints) enable comprehensive optimization from the bottom layer to the top layer.

[0134] Specifically:

[0135] The fusion fidelity loss aims to ensure that the fused image accurately preserves the underlying physical properties of the source image. To balance the differences in the dimensions of each evaluation dimension, the fusion fidelity loss adopts a weighted aggregation form, as follows:

[0136]

[0137] in, This represents the weighting factor for each loss; , , , These are intensity loss, structural similarity loss, color loss, and gradient loss, respectively.

[0138] Among them, intensity fusion target Indicator mask generated from grayscale image comparison Guided generation:

[0139]

[0140] in, This is an indicator function; it outputs 1 if the condition within the parentheses is true, and 0 otherwise.

[0141] The semantic consistency alignment loss constrains the feature distribution orientation of the fused image and the source image in the latent space by sharing a semantic encoder, as follows:

[0142]

[0143] in, Cosine distance; The modal equilibrium coefficient can be set to... =0.5; The decoupling coefficients, used to prevent excessive modal convergence and preserve complementary differences, can be set to... =0.1.

[0144] Quality-guided loss constructs an ideal target by extracting the maximum quality values ​​of the source image in the dimensions of gradient energy, local contrast, and soft entropy. Constrained feedforward prediction score Therefore, we obtain the following details:

[0145]

[0146] Wherein, P is a built-in differentiable evaluation agent, which includes three differentiable evaluation metrics: gradient energy, local contrast, and soft entropy.

[0147] Example 4: This example analyzes and verifies the infrared and visible light image fusion model constructed using the quality-semantic synergy-based infrared and visible light image fusion model construction method, as follows:

[0148] (a) Basic settings

[0149] (1) Data set and comparison method settings

[0150] To fully verify the model's generalization and scene adaptation capabilities, this embodiment employs a two-stage training strategy and conducts comparative experiments on multiple publicly available benchmark datasets. In the training phase, pre-training is first performed using 1083 images from the MSRS dataset to fully learn the general texture, structural, and modal complementarity features of infrared-visible images. Then, incremental fine-tuning training is conducted using 240 images from the M3FD dataset to adapt to the road scene features of the M3FD dataset, improving the model's generalization ability and scene robustness. In the testing phase, four independent datasets with significantly different scenes are selected for evaluation: RoadScene (50 pairs), TNO (30 pairs), M3FD (60 pairs), and LLVIP (250 pairs).

[0151] This embodiment selects nine representative image fusion algorithms as baseline comparison methods, including: TarDAL, U2Fusion, CDDFuse[4], MURF, CrossFuse, EMMA, Text-IF, DCEvo and SAM-based SpTFuse.

[0152] (2) Evaluation indicators

[0153] To objectively quantify the quality of the fused images, this embodiment selects eight multi-dimensional image fusion evaluation metrics, including: difference correlation and quality. and Information entropy Visual information fidelity Edge retention Nonlinear filtering edge degree and image structure-based and For all the evaluation metrics mentioned above, higher values ​​indicate better quality fused images.

[0154] (3) Implementation Details

[0155] The proposed method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration in this embodiment is implemented using the PyTorch deep learning framework. All experiments were performed on a single NVIDIA GeForce RTX 3090 GPU. The network training used the Adam optimizer, with the basic pre-training stage set to 20 epochs, the fine-tuning stage set to 80 epochs, and the batch size set to 8.

[0156] In the downstream object detection evaluation experiments, a standard YOLOv7 detector was used, trained and evaluated based on the official six-class annotations of the M3FD dataset: pedestrians, cars, bicycles, motorcycles, trucks, and buses. The detection network employed a three-scale detection head (P3 / P4 / P5), with 300 epochs of training. The initial learning rate was 0.01, the optimizer was SGD, the momentum coefficient was 0.937, the weight decay was 5e-4, and a 3-epoch warmup strategy was used. Standard data augmentation methods such as Mosaic, MixUp, and random flipping were employed during training. In the testing phase, the input resolution was uniformly adjusted to 640×640, the IoU threshold for NMS was set to 0.65, and the COCO-style mAP metric was used for performance evaluation.

[0157] (ii) Subjective visual comparison

[0158] To visually demonstrate the fusion performance of the method of this invention, the appendix is ​​attached. Figures 8 to 11 The results of fusing different methods in typical scenarios on four datasets: TNO, M3FD, RoadScene, and LLVIP are presented.

[0159] Visual results show that traditional deep learning methods based on feature-level fusion (such as U2Fusion and CrossFuse) are prone to producing halo artifacts around thermal radiation targets, resulting in blurred edges. Methods based on feature decoupling (such as CDDFuse) can improve contrast, but they easily introduce unnatural noise patches in the background area. SpTFuse, based on advanced vision task models, performs stably in terms of semantic saliency, but occasionally causes distortion of local minor textures. In contrast, thanks to the fine modulation at the image level by the Adaptive Semantic Guidance Module (ASG), the model constructed in this invention can effectively overcome the above defects. The fusion result not only clearly preserves the significant thermal radiation characteristics of the infrared target, but also completely restores the scene context and texture details in the visible light image, and the overall visual performance is more in line with human visual perception.

[0160] (III) Objective Quantitative Comparison

[0161] To further verify the effectiveness of the proposed method, this embodiment tested all methods on four datasets using objective metrics. The quantitative results are shown in Tables 1 to 4, where the shaded area represents the optimal value and the bold area represents the suboptimal value.

[0162] Table 1 Objective Quantitative Evaluation Table for TNO Dataset

[0163]

[0164] Table 2 Objective Quantitative Evaluation Table for M3FD Dataset

[0165]

[0166] Table 3 Objective Quantitative Evaluation Table of RoadScene Dataset

[0167]

[0168] Table 4 Objective Quantitative Evaluation Table for the LLVIP Dataset

[0169]

[0170] Quantitative data show that QSFusion demonstrates a significant leading advantage across most evaluation metrics. On the classic TNO dataset (see Table 1), our method achieves [performance / performance]. , , and Achieving the red optimality in indicators such as information entropy ( The accuracy reached 7.0629 and the visual information fidelity ( QSFusion achieved a suboptimal performance for blue when it reached 0.8868. On the M3FD dataset (see Table 2), QSFusion performed exceptionally well, particularly in reflecting the perceptual capabilities of the visual system. Metrics (reaching 1.0456) and edge preservation The red optimum was achieved on the RoadScene dataset (reaching 0.7011). This confirms that the proposed Adaptive Semantic Guidance (ASG) module can effectively capture salient object edges in complex scenes. On the RoadScene dataset (see Table 3), the model constructed in this invention achieved... , , and In terms of key indicators, it consistently ranks first in the red, among which The excellent performance of 7.4367 further demonstrates that the Multi-Scale Interaction (MCI) module can more fully integrate the complementary information of the source image when processing large-scale backgrounds.

[0171] For the LLVIP dataset (see Table 4) in low-light scenes, QSFusion performs well. Reaching 0.7377 and The value reached 0.8197, achieving the optimal value for red, and was also found in the visual structure index. It achieved a leading score of 0.8528. This indicates that the endogenous semantic-driven mechanism can still maintain robust structural feature extraction capabilities under extremely low light conditions.

[0172] Quantitative analysis further supports subjective visual evaluation. In particular, regarding visual information fidelity (… ) and information entropy ( The consistently leading performance across various metrics indicates that the feature distribution of the network output closely matches human visual perception. This is attributed to the guidance of the quality-semantic joint supervision mechanism, which enables the network to utilize high-dimensional semantics to guide the generation of underlying pixels, thereby maintaining high contrast while ensuring the integrity of structural information. Based on the performance across different datasets, the model constructed in this invention demonstrates outstanding scene generalization ability.

[0173] (iv) Evaluation for downstream tasks: target detection

[0174] High-quality image fusion not only pursues excellence in underlying visual metrics, but its core value lies in providing more discriminative and robust feature representations for downstream machine vision tasks. To verify the application value of fused images in real-world complex scenes, this embodiment performs object detection evaluation experiments on the M3FD dataset, which contains abundant road targets. (Appendix) Figure 12 The results of the visualization comparison of different fusion methods are shown, while Table 5 records in detail the precision (P), recall (R), and mean precision (mAP) of each method on “Person”, “Car”, and “All”.

[0175] Table 5. Quantitative comparison of target detection performance in the upstream and downstream of the M3FD dataset.

[0176]

[0177] Experimental data show that the fused images generated by QSFusion provide the most favorable physical feature cues for the detection network, achieving a mean precision (mAP@.5) of 0.963 across all categories. Compared to the baseline method TarDAL's 0.922 and SpTFuse's 0.950 (which incorporates prior knowledge from advanced vision tasks), the model constructed in this invention achieves a stable and significant performance improvement. In terms of recall, the model constructed in this paper achieves an overall recall R of 0.613, significantly outperforming traditional contrast algorithms such as U2Fusion. This strongly demonstrates that fused images with high contrast and rich scene context preservation help the detector more accurately locate potential targets in complex environments, thereby significantly reducing the false negative rate.

[0178] Furthermore, the model constructed in this invention maintains robust accuracy even when dealing with small-scale and feature-variable categories such as "pedestrians." This advantage is mainly attributed to the ASG module's precise enhancement of infrared salient targets and the MCI module's meticulous preservation of multi-scale details. Through a quality-semantic joint supervision mechanism, this invention successfully brings the need for machine perception forward to the image generation stage. Experimental data fully demonstrates that the strategy of using high-dimensional semantic features to guide the generation of low-level pixels can indeed bridge the semantic gap between low-level reconstruction and high-level cognition, raising the upper limit of perception for downstream detection tasks.

[0179] (V) Ablation Experiments and Analysis

[0180] To verify the effectiveness of the core architectural components in the QSFusion network, this invention conducted ablation experiments on the Adaptive Semantic Guidance Module (ASG), Multi-Scale Cross-Dimensional Interaction Module (MCI), and Quality Guidance (QG) branch on the RoadScene and TNO datasets. Tables 6 and 7 show the experimental structure data.

[0181] Table 6. Quantitative Comparison of TNO Dataset Ablation Experiments

[0182]

[0183] Table 7. Quantitative Comparison of Ablation Experiments on the RoadScene Dataset

[0184]

[0185] Based on the experimental data in Tables 6 and 7, the following analytical conclusions can be drawn:

[0186] Effectiveness of the Adaptive Semantic Guidance Module (ASG): Removing the ASG module leads to a significant decrease in most evaluation metrics. Taking the RoadScene dataset as an example... The metric dropped sharply from 0.8096 to 0.4835. This indicates that utilizing Sobel edge information at the input for target augmentation and background compensation provides crucial spatial priors for decoupling the network's target from the background.

[0187] Effectiveness of the Multi-Scale Cross-Dimensional Interaction Module (MCI): Removing the MCI module significantly reduces spatial detail and visual fidelity. For example, in the RoadScene dataset, It decreased from 0.8096 to 0.5530. The value dropped from 0.6161 to 0.2500. This demonstrates that single-dimensional feature stitching lacking cross-dimensional interaction easily leads to degradation of underlying details, while the MCI module effectively achieves synergistic enhancement of high-level semantic cognition and underlying texture details.

[0188] The supplementary role of Quality Guidance (QG): After removing the quality guidance constraints, the network exhibits significant degradation in core metrics such as fidelity and edge structure. In the RoadScene dataset, It dropped to 0.7478. The value dropped to 0.5103. This confirms that the quality-semantic joint optimization mechanism plays an indispensable supporting role in maintaining the stability of the global image structure and guiding the accurate reconstruction of low-level features.

[0189] Based on the above analysis and verification, this invention aims to alleviate the problem of balancing low-level visual fidelity and high-level semantic cognition in traditional image fusion, and proposes a QSFusion network that does not rely on high-level visual task models. Experiments show that QSFusion exhibits good overall performance in both subjective and objective evaluations on four benchmark datasets. It not only achieves high-quality low-level visual fidelity, but also provides highly discriminative feature cues for downstream object detection tasks, significantly improving the accuracy of object detection.

[0190] Example 5: As shown in the attached document Figure 13 As shown, the infrared and visible light image fusion method includes:

[0191] Step S210: Obtain a visible light image and an infrared image to be fused;

[0192] Step S220: Input a visible light image and an infrared image to be fused into the infrared and visible light image fusion model to obtain the corresponding fused image. The infrared and visible light image fusion model is constructed by the infrared and visible light image fusion model construction method based on quality-semantic collaboration.

[0193] Example 6: As attached Figure 14 As shown, the device for constructing an infrared and visible light image fusion model based on quality-semantic collaboration includes:

[0194] The sample acquisition unit acquires several samples and divides them into a training sample set and a test sample set according to a ratio. Each sample includes a visible light image and an infrared image to be fused, as well as the corresponding fused image.

[0195] The training unit trains the QSFusion network using a training sample set. Training ends when the total loss function stabilizes, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulates it to obtain infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention-weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image.

[0196] The testing unit uses a test sample set to perform performance testing and quantitative evaluation on the trained infrared and visible light image fusion models, and selects infrared and visible light image fusion models that meet the evaluation criteria.

[0197] In this embodiment, the specific implementation steps of each unit are as shown in the method embodiment, and will not be repeated here.

[0198] Example 7: As attached Figure 15 As shown, the infrared and visible light image fusion apparatus applying the above-described infrared and visible light image fusion method includes:

[0199] The basic data acquisition unit acquires a visible light image and an infrared image to be fused.

[0200] The fusion unit inputs a visible light image and an infrared image to be fused into the infrared and visible light image fusion model to obtain the corresponding fused image.

[0201] Example 8: The electronic device that executes the above-described method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration or the infrared and visible light image fusion method includes a processor and a memory. The memory stores a computer program, which is loaded and executed by the processor to realize the method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration or the infrared and visible light image fusion method.

[0202] The processor described above can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. It can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The memory can include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, portable hard drives, magnetic disks, or optical disks.

[0203] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0204] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] The above content is only a specific embodiment of the present invention, which has strong adaptability and implementation effect. However, the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention. Therefore, equivalent changes made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for constructing an infrared and visible image fusion model based on quality-semantic synergy, characterized in that, include: Several samples were obtained and divided into a training sample set and a test sample set according to the proportion. Each sample included a visible light image and an infrared image to be fused, as well as the corresponding fused image. The QSFusion network was trained using a training sample set. Training ended when the total loss function stabilized, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulate it to obtain infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention-weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image. The performance of the trained infrared and visible light image fusion model was tested and quantitatively evaluated using a test sample set, and infrared and visible light image fusion models that meet the evaluation criteria were selected.

2. The method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration according to claim 1, characterized in that, The process by which the adaptive semantic guidance module generates a spatially aware mask includes: The Sobel operator was used to extract the high-frequency edge response of the grayscale components of infrared and visible light images, respectively. Two high-frequency edge responses are spatially semantically integrated through two independent hierarchical convolutional branches to obtain two output features; After acquiring the two output features, the features are concatenated along the channel dimension, and a spatially aware mask is generated by fusing two steps of convolution and compressed convolution.

3. The method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration according to claim 1 or 2, characterized in that, The dual-stream coding unit includes an infrared encoder and a visible light encoder. Both the infrared encoder and the visible light encoder consist of four consecutive Transformer modules. Each Transformer module includes a multi-head depth transposed attention network and a convolutional feedforward network.

4. The method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration according to claim 1 or 2, characterized in that, The deep interaction fusion unit includes an attention weighting value aggregation operation module, a multi-scale cross-dimensional interaction module, and a two-way cross attention module; Two cross-attention modules are used to perform preliminary alignment of dual-modal features for infrared high-dimensional endogenous semantic features and visible light high-dimensional endogenous semantic features through a cross-attention mechanism. The attention-weighted aggregation operation module is used to perform feature attention-weighted aggregation on the aligned features. Then, the aggregated features are adaptively weighted and optimized through the attention-weighted aggregation operation to achieve deep interaction and representation enhancement of multimodal features. Finally, the enhanced features are output through residual connections. By utilizing a multi-scale, cross-dimensional interactive module, the enhanced features are deeply enhanced to obtain collaborative features.

5. The method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration according to claim 1 or 2, characterized in that, The decoding and thinning output unit includes a decoder, a thinning module, an output convolutional block, and a Sigmoid activation function; The decoder uses four consecutive Transformer modules to gradually recover the spatial resolution and obtain the reconstructed features; The reconstructed features are processed using a refinement module to eliminate residual artifacts; After eliminating residual artifacts, a fused image is generated by applying the output convolutional block and the Sigmoid activation function.

6. The method for constructing an infrared and visible light image fusion model based on quality-semantic collaboration according to any one of claims 1 to 5, characterized in that, The total loss function is as follows: in, To compensate for the loss of fidelity; This is the semantic consistency alignment loss; Loss guided by quality; This is the regularization loss; , , All of these are weighting coefficients for each type of loss.

7. A method for fusing infrared and visible light images, characterized in that, include: Acquire a visible light image and an infrared image to be fused; A visible light image and an infrared image to be fused are input into an infrared and visible light image fusion model to obtain a corresponding fused image, wherein the infrared and visible light image fusion model is constructed by the method described in any one of claims 1 to 6.

8. An apparatus for constructing an infrared and visible light image fusion model based on quality-semantic collaboration, applying the method described in any one of claims 1 to 6, characterized in that, include: The sample acquisition unit acquires several samples and divides them into a training sample set and a test sample set according to a ratio. Each sample includes a visible light image and an infrared image to be fused, as well as the corresponding fused image. The training unit trains the QSFusion network using a training sample set. Training ends when the total loss function stabilizes, resulting in an infrared and visible light image fusion model. The QSFusion network includes: an input modulation unit that introduces an adaptive semantic guidance module to generate a spatially aware mask and modulates it to obtain infrared and visible light image enhancement features; a dual-stream coding unit that captures cross-scale high-dimensional endogenous semantic features from the infrared and visible light image enhancement features; a deep interactive fusion unit that introduces an attention-weighted aggregation operation module and a multi-scale cross-dimensional interaction module to fuse and enhance the input high-dimensional endogenous semantic features to obtain collaborative features; and a decoding and refinement output unit that decodes, reconstructs, and refines the collaborative features to output the corresponding fused image. The testing unit uses a test sample set to perform performance testing and quantitative evaluation on the trained infrared and visible light image fusion models, and selects infrared and visible light image fusion models that meet the evaluation criteria.

9. An infrared and visible light image fusion apparatus applying the method as described in claim 7, characterized in that, include: The basic data acquisition unit acquires a visible light image and an infrared image to be fused. The fusion unit inputs a visible light image and an infrared image to be fused into the infrared and visible light image fusion model to obtain the corresponding fused image.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 6 or to implement the steps of the method as claimed in claim 7.

Images