Image fusion method, model and storage medium
By employing a heterogeneous dual-branch model and cross-attention fusion technology, the problems of dynamic range imbalance and brightness deviation in the fusion of stained and collagen images were solved, achieving high-quality image fusion and tissue analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies struggle to simultaneously preserve the morphological information of stained images and the collagen-specific information of collagen images within a unified framework. This results in dynamic range imbalance, input source brightness deviation, discrepancies between quality analysis indicators and practical application value, and insufficient generalization ability in fused images.
A heterogeneous dual-branch model is adopted, which separates the training set and the test set to extract local detail features of the stained image and global context features of the collagen image respectively. The model is then weighted and fused using a cross-attention fusion module, and the model training is optimized using a combined loss function to ensure feature extraction and reconstruction across different modalities.
It achieves high-fidelity feature fusion across different modalities, solves the problems of dynamic range imbalance and brightness deviation, improves the generalization ability of the model and the quality analysis indicators of the fused image, and provides a more comprehensive view of tissue analysis.
Smart Images
Figure CN121746205A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically, to an image fusion method, model, and storage medium. Background Technology
[0002] In medicine, stained images and collagen images are often used to analyze tissues. Stained images can clearly display key information such as cell morphology and tissue structure, but they lack specific imaging of the matrix components of the tissue microenvironment. Collagen images can specifically capture the fine texture and spatial distribution of collagen fibers. Therefore, fusing the morphological information of stained images with the collagen-specific information of collagen images can provide a more comprehensive view for tissue analysis.
[0003] However, the morphological details such as cell nuclei in stained images are extremely fine and easily disturbed, while the collagen texture in collagen images exhibits complex multi-scale and global characteristics. Current fusion methods struggle to preserve these two vastly different features simultaneously within a unified framework, leading to problems such as dynamic range imbalance, input source brightness deviation, discrepancies between quality analysis metrics and practical application value, and insufficient generalization ability in fused images, thus failing to meet the needs of practical applications. Summary of the Invention
[0004] The purpose of this invention is to solve the above problems and provide an image fusion method, model and storage medium, which is a multimodal image fusion method that can adaptively balance morphological information and collagen-specific information and has high generalization ability.
[0005] This invention provides the following technical solution: Firstly, an image fusion method includes Obtain datasets of stained and collagen images, which are divided into training and testing sets; The training set includes a group of stained images and a group of collagen images; The test set includes multiple pairs of stained images and collagen images; The training set is used to train a pre-defined heterogeneous dual-branch model, with each training session using either a set of stained images or a set of collagen images. The paired stained images and collagen images in the test set are input and processed by the trained heterogeneous dual-branch model to obtain the fused image.
[0006] In relevant embodiments, the training set trains a preset heterogeneous dual-branch model, and each training session uses either a stained image set or a collagen image set, including: During training, either the stained image group or the collagen image group is randomly selected; The training images are used to extract local detail features and global context features in a pre-defined heterogeneous dual-branch model, respectively. Local detail features and global context features are reconstructed by feature fusion in a pre-defined heterogeneous dual-branch model, and the fused image is output.
[0007] In relevant embodiments, the paired stained images and collagen images from the test set are input through a trained heterogeneous dual-branch model to obtain a fused image, including: Paired stained images and collagen images from the test set are input into the trained heterogeneous dual-branch model to extract local detail features and global context features, respectively, to obtain local features of stained images and global context features of collagen images. The final fusion features are determined by combining local features of the stained image and global contextual features of the collagen image. The final fusion features are used to reconstruct the fused image.
[0008] In relevant embodiments, the final fusion features are determined by combining local features of the stained image and global contextual features of the collagen image, including: Local features of the stained image and global contextual features of the collagen image are fused to obtain the overall features of the stained image and the overall features of the collagen image, respectively. The final fusion features are determined by weighted summation based on the overall features of the stained image and the overall features of the collagen image.
[0009] In relevant embodiments, the final fusion features are determined by weighted summation based on the overall features of the stained image and the overall features of the collagen image, including: The weights of the stained image are determined based on the overall features of the stained image, the overall features of the collagen image, the L1 norm, and the L2 norm. The weights of the collagen images are determined based on the weights of the stained images; The final fusion features are determined by weighted summation based on the weights of the stained image, the collagen image, the overall features of the stained image, and the overall features of the collagen image.
[0010] The training set is used to train a pre-defined heterogeneous dual-branch model using a combined loss function, which is determined based on the mean squared error loss function, the structural similarity loss function, and the total variational loss function.
[0011] Secondly, an image fusion model includes Heterogeneous dual-branch encoder: includes a densely connected convolutional branch module for extracting local detail features of the image, and a pyramid attention branch module for extracting global context features of the image; The pyramid attention branch module includes a hollow spatial pyramid pooling module for convolutional pooling and fusion of images, and a dual-window multi-head attention module for simultaneously capturing local fine-grained features and global contextual information.
[0012] Cross-attention fusion module: It connects to the output of the heterogeneous dual-branch encoder and is used to adaptively weight and fuse the dual-modal features output by the dual-branch encoder in spatial and channel dimensions; Decoder: It is connected to the output of the cross-attention fusion module and uses the features output by the cross-attention fusion module to reconstruct and generate a fused image.
[0013] In a second aspect, a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an image fusion method.
[0014] Thirdly, an electronic device includes a memory and a processor, the memory storing a computer program executable on the processor, the processor executing the computer program to implement the steps of an image fusion method.
[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention divides the dataset of stained images and collagen images into a training set and a test set; the paired stained images and collagen images in the test set are input through the trained heterogeneous dual-branch model to obtain a fused image; after extracting morphological details such as cell nuclei from the stained images and collagen information from the collagen images respectively through the heterogeneous dual-branch model, the fused image is obtained by fusing them. The use of a heterogeneous dual-branch model instead of a unified framework for feature extraction of the two types of images solves the problems of dynamic range imbalance, input source brightness deviation, difference between quality analysis indicators and practical application value, and insufficient generalization ability of the related fused images.
[0016] (2) In this invention, the training set includes a stained image set and a collagen image set. The training set trains a pre-defined heterogeneous bi-branch model. Each training session uses either the stained image set or the collagen image set. The single-modality image input means that no matter what modality is input, the heterogeneous bi-branch model must extract its essential structural features to complete accurate reconstruction. This drives the heterogeneous bi-branch model to work together and converge to a shared feature space, thereby effectively preventing overfitting to a single-modality image, significantly improving the model's generalization ability, and laying a solid foundation for high-quality bi-modal fusion in the inference stage. Attached Figure Description
[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic diagram of the overall process structure of this embodiment.
[0018] Figure 2 This is a schematic diagram of the heterogeneous dual-branch model training process in this embodiment.
[0019] Figure 3 This is a schematic diagram of the inference process structure of the heterogeneous dual-branch model in this embodiment.
[0020] Figure 4 This is a schematic diagram of the process structure for obtaining the final fusion features based on weights in this embodiment.
[0021] Figure 5 This is a schematic diagram of the heterogeneous dual-branch model training phase in this embodiment.
[0022] Figure 6 This is a schematic diagram of the inference stage structure of the heterogeneous dual-branch model in this embodiment.
[0023] Figure 7 This is a schematic diagram of the densely connected convolutional branch structure in this embodiment.
[0024] Figure 8 This is a schematic diagram of the decoder structure in this embodiment.
[0025] Figure 9 This is a schematic diagram of the pyramid attention branch module structure in this embodiment.
[0026] Figure 10 This is a schematic diagram of the cross-attention fusion module structure in this embodiment.
[0027] Figure 11 This is a schematic diagram of the spatial attention module structure in this embodiment.
[0028] Figure 12 This is a comparison diagram of the fusion method for changes in the infiltrated catheter structure in this embodiment.
[0029] Figure 13 This is a comparative diagram of the deep infiltration and fusion method of bladder tissue in this embodiment.
[0030] Figure 14 This is a comparative diagram of the deep infiltration and fusion method of esophageal tissue in this embodiment. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the following will be combined with... Figures 1 to 14 The present invention will be described in further detail below. The described embodiments should not be regarded as limitations on the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] To achieve high-quality feature extraction and effective fusion of stained and collagen images, providing a more comprehensive view for tissue analysis, and addressing issues such as dynamic range imbalance, input source brightness deviation, discrepancies between quality analysis indicators and practical application value, and insufficient generalization ability in related fusion methods.
[0033] like Figure 1 As shown, this embodiment provides an image fusion method including: S100. Obtain the dataset of stained images and collagen images, which are divided into training set and test set; stained images include fused hematoxylin-eosin stained images (H&E), and collagen images include second harmonic generation collagen images (SHG). Common digital image formats are supported, including but not limited to TIFF, PNG, JPEG, etc.
[0034] S200, the training set includes a group of stained images and a group of collagen images; S300, the test set includes multiple pairs of stained images and collagen images; S400, the training set is used to train a pre-defined heterogeneous dual-branch model, and each training session uses either a group of stained images or a group of collagen images. The S500, paired staining images and collagen images from the test set are input and processed by the trained heterogeneous dual-branch model to obtain a fused image.
[0035] This embodiment fundamentally solves the problems of dynamic range imbalance and input source brightness deviation by using a heterogeneous dual-branch encoder and a cross-attention fusion module in the heterogeneous dual-branch model. This technical mechanism ensures that the fused image can simultaneously retain high-fidelity H&E morphological information and SHG collagen information, such as... Figures 12 to 14 The experimental results clearly reproduced key information such as changes in collagen fiber arrangement and tissue structure characteristics, while the contrast methods suffered from severe overexposure or insufficient contrast. The contrast methods included MSVD, VSMWLS, SDNet, IFCNN, and SwinFusion. Staining images, such as the fused hematoxylin-eosin staining image H&E, and collagen images, such as the second harmonic generation collagen image SHG, were also included. Figures 12 to 14 Chinese Glossary: H&E: Fusion hematoxylin-eosin staining images.
[0036] H&E (ROI): The content within the white box in the H&E diagram.
[0037] SHG: Second harmonic generation of collagen images.
[0038] SHG (ROI): The content within the white box in the SHG chart.
[0039] MSVD: A traditional fusion algorithm based on multi-scale singular value decomposition.
[0040] VSMWLS: An image fusion method that combines visual saliency maps and weighted least squares optimization.
[0041] SDNet: A general deep learning fusion network based on autoencoders.
[0042] IFCNN: A convolutional neural network designed specifically for image fusion.
[0043] SwinFusion: abbreviated as Swin, is an advanced fusion model based on the Swin Transformer architecture.
[0044] To address the discrepancy between relevant analytical metrics and practical application value, this embodiment proposes a decompositional analysis method. This method decomposes the fusion quality into two independent dimensions: similarity to the basic morphological information of H&E and similarity to the specific collagen texture of SHG. This strategy avoids the limitation of traditional metrics over-reliance on global brightness features and can more accurately reflect the value of the fused image in practical applications.
[0045] like Figure 2 and 5 As shown, the training set trains a pre-defined heterogeneous dual-branch model. Each training session uses either a stained image set or a collagen image set, including: S410. Randomly select either the stained image group or the collagen image group during training. S420. The training image extracts local detail features and global context features respectively in the preset heterogeneous dual-branch model; S430, local detail features, and global context features are reconstructed after feature fusion in a preset heterogeneous dual-branch model, and the fused image is output.
[0046] This embodiment employs a training-inference separation model. This design effectively solves the training instability and modality bias problems caused by the inherent differences in bimodal inputs during training in traditional end-to-end fusion models, and enables the learning of a unified, modality-independent feature representation. Specifically: Training Phase: An autoencoder reconstruction strategy is employed, randomly selecting either H&E or SHG single-modality images as input in each training batch. During training, each input image is processed simultaneously through two branches: a densely connected convolutional branch learns to extract local detail features, while a dilated spatial pyramid-dual-window multi-head attention branch learns to extract global contextual features. The outputs of the two branches are fused using a cross-attention fusion module, and finally, the input image is reconstructed through a decoder. The training objective is to minimize the difference between the input image and the reconstructed image.
[0047] The core idea is to force the model to learn a general feature representation that transcends specific modalities through an autoencoder reconstruction task: regardless of the input modality, the encoder must extract its essential structured features to complete accurate reconstruction. This drives the heterogeneous dual-branch encoder to work together and converge to a shared feature space, thereby effectively preventing overfitting to a single modality, significantly improving the model's generalization ability, and laying a solid foundation for high-quality dual-modal fusion in the inference stage.
[0048] Hyperparameter configuration: Batch size: 8 Initial learning rate: 0.001 Number of training rounds: 50 Learning rate decay strategy: exponential decay, decay rate 0.9 like Figure 6 As shown, in the inference phase: Paired H&E and SHG images are input into the trained network for feature extraction and fusion. Thanks to the powerful and unified feature representation learned during training, the bimodal images are mapped into a compatible feature space, enabling the Cross-Attention Fusion Module (CAFM) and adaptive weighting strategy to accurately mine and fuse complementary information, thereby preserving both the morphological details of H&E and the collagen texture features of SHG in the final fused image with high fidelity.
[0049] The innovative training-inference separation mode, multi-scale feature capture module, and loss function optimized for medical images work together to ensure that the model can generate high-quality fused images across a variety of tissue types, demonstrating excellent generalization ability.
[0050] like Figure 3 As shown, the paired stained images and collagen images in the test set are input through a trained heterogeneous dual-branch model to obtain fused images, including: S510. Input the paired stained images and collagen images in the test set into the trained heterogeneous dual-branch model to extract local detail features and global context features respectively, and obtain the local features of the stained images and the global context features of the collagen images. S520. Determine the final fusion features by combining local features of the stained image and global contextual features of the collagen image; S530. The final fusion features are used to reconstruct the fused image.
[0051] like Figure 4 As shown, in the relevant embodiments, the final fusion features are determined by using local features of the stained image and global contextual features of the collagen image, including: S521. Local features of the stained image and global contextual features of the collagen image are fused to obtain the overall features of the stained image and the overall features of the collagen image, respectively. S522. Determine the weights of the stained image based on the overall features of the stained image, the overall features of the collagen image, the L1 norm, and the L2 norm. S523. Determine the collagen image weights based on the weights of the stained images; S524. Based on the weights of the stained image, the collagen image, the overall features of the stained image, and the overall features of the collagen image, the weighted sum is used to determine the final fusion features.
[0052] Example: 1. Bimodal Independent Encoding: H&E and SHG images are input into a pre-trained encoder network with shared weights. This encoder network includes a heterogeneous dual-branch and cross-attention fusion module. 2. Branch Feature Extraction and Intramodal Fusion: For each input image, the encoder network extracts features through densely connected convolutional branches and dilated spatial pyramid-dual-window multi-head attention branches, respectively, and performs intramodal feature fusion through a cross-attention fusion module to obtain the overall feature representation of the H&E image. Overall feature representation of SHG images ; 3. Intermodal adaptive fusion: The fusion weights of the two modal features are calculated based on the L1-L2 norm, and the final fused features are obtained by weighted summation. Specifically, it includes: Calculate the fusion weights of H&E modal features : in, This represents the feature tensor extracted from the H&E image. This represents the feature tensor extracted from the SHG image; and This represents the L1 norm and L2 norm calculated along the channel dimension. This indicates that the mean value is calculated over the spatial dimension.
[0053] Calculate the fusion weights of SHG modal features : The final fusion feature is obtained by weighted summation: 4. Image reconstruction: Input the final fusion features into the decoder to generate a fused image.
[0054] In relevant embodiments, the training set trains a pre-defined heterogeneous dual-branch model using a combined loss function, which is determined based on the mean squared error loss function, the structural similarity loss function, and the total variational loss function.
[0055] Example: The training of the image fusion model uses a combination loss function, the expression of which is: in, For the total loss, For mean square error loss, For structural similarity loss, For the total variational loss, and This is the weighting coefficient, with a preferred value of 100.
[0056] In the combined loss function, the mean squared error loss focuses on pixel-level accurate matching, ensuring the consistency between the fused image and the target image at the pixel level; the structural similarity loss focuses on visual perception quality, effectively preserving the structural information and texture features of the image; and the total variational loss focuses on the spatial smoothness of the image, helping to reduce noise and artifacts while maintaining edge and texture details. By combining these three loss functions, the pixel-level accuracy, visual perception quality, and spatial smoothness of the image can be optimized simultaneously during model training, thereby obtaining fused image results with higher application value.
[0057] A systematic comparison with five representative methods (MSVD, VSMWLS, SDNet, IFCNN, and SwinFusion) shows that our proposed method significantly improves the generation quality of SHG collagen texture while fully preserving H&E morphological information. Specifically, in terms of structural similarity SHG metrics, our method achieves macroscopic average and weighted average scores of [insert scores here]. Compared to the best baseline model, the improvement is 1.30 to 1.43 times. In large-scale tests covering multiple tissue types, this method showed stable and significant performance advantages on most tissue types.
[0058] This method has been thoroughly validated on a large-scale, multicenter dataset containing multiple tissue types and over 10,000 paired image patches, demonstrating its high reliability and stability in practical applications.
[0059] This embodiment also provides an image fusion model including: Heterogeneous dual-branch encoder: includes a densely connected convolutional branch module for extracting local detail features of the image, and a pyramid attention branch module for extracting global context features of the image; This encoder employs a heterogeneous dual-branch design, designed to adapt to the morphological characteristics of stained images and the texture and multi-scale contextual characteristics of collagen images, respectively. During training, the two branches jointly learn to extract complementary feature representations of the input images, providing a foundation for subsequent fusion.
[0060] like Figure 7As shown, densely connected convolutional branches: Network structure: It contains 5 convolutional layers and adopts a dense connection mechanism. The specific parameter configuration is shown in the table below:
[0061] Technical effect: By using a dense connection mechanism, the features of the previous layer are concatenated with the output of the current layer in the channel dimension to achieve feature reuse, effectively alleviate the gradient vanishing problem in deep networks, and enhance the ability to extract local detail features of images. It is particularly suitable for preserving morphological features such as cell nuclei and tissue structures.
[0062] The pyramid attention branch module includes a hollow spatial pyramid pooling module for convolutional pooling and fusion of images; it also includes a dual-window multi-head attention module for simultaneously capturing local fine-grained features and global contextual information.
[0063] Pyramid attention branch module: such as Figure 9 As shown, an integrated dilated convolution and dual-window self-attention mechanism is used to capture multi-scale global contextual information and collagen texture features of collagen images: like Figure 10 As shown, the Aperture Space Pyramid Pooling Module (ASPP): Structure: It consists of a 1×1 standard convolutional layer and three 3×3 dilated convolutional layers with different dilation rates (6, 12 and 18 respectively). Each convolutional layer has 80 output channels, as well as a global average pooling branch.
[0064] Processing flow: The input image is processed in parallel through 1×1 convolution, three dilated convolutions with different dilation rates, and a global average pooling branch. The outputs of each branch are concatenated along the channel dimension and then fused through 1×1 convolution to output a 256×256×80 feature map, effectively capturing multi-scale contextual information.
[0065] like Figure 11 As shown, the parameters of the Dual-Window Multi-Head Attention Module (DWMAB) are configured as follows:
[0066] Technical benefits: By using a dual-window design to simultaneously capture local fine-grained features and global contextual information, and using a multi-head attention mechanism to achieve feature representation in multiple subspaces, it is particularly suitable for extracting complex texture features and multi-scale structures, such as biological tissue structures like collagen fibers.
[0067] Cross-attention fusion module: It connects to the output of the heterogeneous dual-branch encoder and is used to adaptively weight and fuse the dual-modal features output by the dual-branch encoder in spatial and channel dimensions; Cross-Attention Fusion Module (CAFM) Connecting the output of the dual-branch encoder, adaptive weighted fusion of the bimodal features is performed in both spatial and channel dimensions: Module composition: Two independent channel attention modules (CAMs) are used, with a dimensionality reduction ratio of 16. Two independent spatial attention modules (SAM) with a kernel size of 3×3; A 1×1 fused convolutional layer with 80 output channels; Processing flow: Channel attention interaction: Calculate the channel attention weights of the two input features respectively, construct the channel correlation matrix, and realize the remapping of features in the other channel space through matrix multiplication to explore the semantic association between the two modal features.
[0068] Spatial attention enhancement: Apply spatial attention mechanism to the interactive features to generate a spatial importance weight map, and maintain the original feature information through residual connections to achieve adaptive selection of spatial location.
[0069] Feature fusion output: The enhanced dual-modal features are concatenated along the channel dimension, and then dimensionality reduction and final fusion are performed through 1×1 convolution to output a fused feature map of 256×256×80.
[0070] (3) Decoder: It is connected to the output of the cross-attention fusion module and uses the features output by the cross-attention fusion module to reconstruct and generate a fused image.
[0071] decoder like Figure 8 As shown, it consists of five consecutive convolutional layers. The final fused image is generated by reconstructing the fused features. Specific parameter configurations:
[0072] Technical features: All convolutional layers employ reflection padding to preserve feature map size and avoid boundary artifacts; the last layer uses the tanh activation function to limit the output value to the range [-1, 1]. This design ensures that the generated fused image can simultaneously and clearly present the cell nuclear morphology in H&E staining and the collagen fiber structure in SHG images, providing more comprehensive information for tissue analysis.
[0073] Experimental results, along with a systematic comparison with five representative methods (MSVD, VSMWLS, SDNet, IFCNN, and SwinFusion), demonstrate that our proposed method significantly improves the generation quality of SHG collagen textures while fully preserving H&E morphological information. Specifically, in terms of structural similarity (SHG) metrics, our method achieves [specific scores - likely 100%] for both the macroscopic average and weighted average. Compared to the best baseline model, the improvement is 1.30 to 1.43 times. In large-scale tests covering multiple tissue types, this method showed stable and significant performance advantages on most tissue types.
[0074] This embodiment has been thoroughly validated using a large-scale, multi-center dataset containing multiple tissue types and over ten thousand paired image patches, demonstrating its high reliability and stability in practical applications.
[0075] This embodiment also provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of an image fusion method.
[0076] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, it implements the steps of an image fusion method.
[0077] Electronic devices can be desktop computers, laptops, handheld computers, and cloud servers, among other electronic devices. Electronic devices may include, but are not limited to, processors and memory. Those skilled in the art will understand that the figures are merely examples of electronic devices and do not constitute a limitation on the electronic device; it may include more or fewer components than illustrated, or different components.
[0078] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0079] Memory can be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory can also be an external storage device of an electronic device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Memory can also include both internal and external storage units. Memory is used to store computer programs and other programs and data required by the electronic device.
[0080] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0081] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0082] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0084] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. An image fusion method, characterized in that, include Obtain datasets of stained and collagen images, which are divided into training and testing sets; The training set includes a group of stained images and a group of collagen images; The test set includes multiple pairs of stained images and collagen images; The training set is used to train a pre-defined heterogeneous dual-branch model, with each training session using either a stained image set or a collagen image set. The paired stained images and collagen images in the test set are input and processed by the trained heterogeneous dual-branch model to obtain the fused image.
2. The method according to claim 1, characterized in that, The training set is used to train a pre-defined heterogeneous two-branch model. Each training session uses either a stained image set or a collagen image set, including: During training, either the stained image group or the collagen image group is randomly selected; The training images are used to extract local detail features and global context features in a pre-defined heterogeneous dual-branch model, respectively. Local detail features and global context features are reconstructed by feature fusion in a pre-defined heterogeneous dual-branch model, and the fused image is output.
3. The method according to claim 1, characterized in that, The paired stained and collagen images from the test set are input and processed by the trained heterogeneous dual-branch model to obtain fused images, including: Paired stained images and collagen images from the test set are input into the trained heterogeneous dual-branch model to extract local detail features and global context features, respectively, to obtain local features of stained images and global context features of collagen images. The final fusion features are determined by combining local features of the stained image and global contextual features of the collagen image. The final fusion features are used to reconstruct the fused image.
4. The method according to claim 3, characterized in that, The final fusion features were determined by combining local features of the stained image and global contextual features of the collagen image, including: Local features of the stained image and global contextual features of the collagen image are fused to obtain the overall features of the stained image and the overall features of the collagen image, respectively. The final fusion features are determined by weighted summation based on the overall features of the stained image and the overall features of the collagen image.
5. The method according to claim 4, characterized in that, Based on the overall features of the stained image and the overall features of the collagen image, the final fusion features are determined by weighted summation, including: The weights of the stained image are determined based on the overall features of the stained image, the overall features of the collagen image, the L1 norm, and the L2 norm. The weights of the collagen images are determined based on the weights of the stained images; The final fusion features are determined by weighted summation based on the weights of the stained image, the collagen image, the overall features of the stained image, and the overall features of the collagen image.
6. The method according to claim 1, characterized in that, The training set is used to train a pre-defined heterogeneous dual-branch model using a combined loss function, which is determined based on the mean squared error loss function, the structural similarity loss function, and the total variational loss function.
7. An image fusion model, characterized in that, include Heterogeneous dual-branch encoder: includes a densely connected convolutional branch module for extracting local detail features of the image, and a pyramid attention branch module for extracting global context features of the image; The pyramid attention branch module includes a hollow spatial pyramid pooling module for convolutional pooling and fusion of images, and a dual-window multi-head attention module for simultaneously capturing local fine-grained features and global contextual information. Cross-attention fusion module: It connects to the output of the heterogeneous dual-branch encoder and is used to adaptively weight and fuse the dual-modal features output by the dual-branch encoder in spatial and channel dimensions; Decoder: It is connected to the output of the cross-attention fusion module and uses the features output by the cross-attention fusion module to reconstruct and generate a fused image.
8. A storage medium, characterized in that, It stores a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When a processor executes a computer program, it implements the steps of the method described in any one of claims 1 to 6.