A medical image fusion method based on multi-scale adaptive convolution and transformer

Abstract

The application discloses a medical image fusion method based on multi-scale adaptive convolution and a transformer, and belongs to the field of medical image processing. The method comprises the following steps: in view of the problems of insufficient utilization of multi-scale information and difficulty in balancing structure and content in fusion of fluorescent protein images and phase difference microscopic images, a fusion network for parallel modeling of multi-scale features is constructed; different scale features are parallelly modeled through the network structure, adaptive convolution is introduced to enhance the local detail and global semantic modeling capability, and a transformer is combined to capture long-range dependency; meanwhile, content loss and structure loss are designed to jointly constrain the fusion result. The application effectively improves the contrast, brightness and detail expression capability of the fusion image while ensuring the consistency of the structure, and is suitable for medical microscopic imaging analysis scenes.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence technology, specifically involving a medical image fusion method based on multi-scale adaptive convolution and Transformer. Background Technology

[0002] In the fields of cell biology, molecular biology, and medical microscopy, microscopic imaging techniques are crucial for studying cell structure, protein distribution, and physiological function. Among these, fluorescent protein microscopy and phase-contrast microscopy are two of the most widely used imaging methods with distinctly complementary characteristics. Fluorescent protein microscopy images can generate fluorescence signals through excitation at specific wavelengths, reflecting the functional distribution information of proteins, molecules, or subcellular structures within cells, and are of significant value in protein localization and gene expression analysis. Meanwhile, phase-contrast microscopy images can convert phase changes caused by transparent samples into intensity changes, thus clearly presenting morphological information such as cell outlines and organelle structures, offering advantages such as high resolution and rich structural detail.

[0003] However, due to differences in imaging mechanisms, single-modal microscopic images often struggle to simultaneously capture both functional and structural information. For example, fluorescent protein images typically suffer from low spatial resolution, blurred structural details, and significant background noise, while phase-difference images, although rich in structural information, cannot directly reflect the functional distribution of specific proteins or molecules within cells. This informational complementarity has prompted researchers to utilize image fusion techniques to integrate effective information from multimodal microscopic images, thereby generating fused images that possess both structural integrity and functional expression capabilities to meet the needs of refined cell analysis and quantitative research.

[0004] Existing microscopic image fusion methods can be broadly categorized into two types: traditional methods and deep learning-based methods. Traditional image fusion methods are mostly based on multi-scale transformations, sparse representations, or manually designed fusion rules, such as pyramid transforms, wavelet transforms, non-subsampled contourlet transforms, and sparse representations. These methods typically rely on human experience to design feature extraction and fusion strategies, making it difficult to adapt to different image content and imaging conditions. Furthermore, when processing complex microscopic images, they often suffer from high computational complexity, unstable fusion results, and difficulty in simultaneously capturing multi-scale structural and detail information.

[0005] With the development of deep learning technology, image fusion methods based on convolutional neural networks and generative adversarial networks have gradually become a research hotspot. These methods automatically learn the feature mapping relationships between multimodal images in an end-to-end manner, improving the quality of fused images to some extent. However, existing deep learning fusion methods still have significant shortcomings in the task of fusing fluorescent proteins with phase-difference microscopic images: on the one hand, some methods employ adversarial training mechanisms, leading to unstable training processes and potential problems such as mode collapse or inconsistent fusion results; on the other hand, some convolutional neural network-based methods focus on local feature modeling, making it difficult to fully capture cross-scale contextual information and long-range dependencies in microscopic images, resulting in a difficulty in achieving a balance between structural consistency and content integrity in the fusion results.

[0006] Furthermore, existing methods still fall short in utilizing multi-scale information, often employing single-scale or shallow multi-scale structures, failing to fully extract the detailed features and semantic information contained in microscopic images at different spatial scales. Simultaneously, in terms of loss function design, most methods only constrain at the pixel level, neglecting the consistency of information at the structural and region levels, easily leading to structural distortion or loss of key information in the fused image.

[0007] Therefore, there is an urgent need for a new method for fusing fluorescent proteins with phase difference microscopic images, which can fully explore multi-scale feature information while ensuring the stability of model training, introduce an effective modeling mechanism for global semantic relationships, and jointly constrain the fusion results at the pixel level and the structural level, thereby achieving high-quality fusion of functional and structural information. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a medical image fusion method based on multi-scale adaptive convolution and Transformer, so as to fully utilize multi-scale information and take into account both structural consistency and content integrity during the fusion process.

[0009] The objective of this invention is achieved through the following technical solution: a medical image fusion method based on multi-scale adaptive convolution and Transformer, comprising the following steps: The original medical source images to be fused are obtained, including grayscale medical images and color medical images. When the source image is a color medical image, it is converted from the RGB color space to the YCbCr color space. The Y channel in the YCbCr color space is extracted as the brightness information input to the network, and the Cb and Cr channels are used for color reconstruction. A multi-scale image fusion network is constructed, which includes several parallel branches of different scales. Each branch consists of at least one adaptive convolutional module and several Transformer modules connected in series. The original medical source image is input into a multi-scale image fusion network to extract multi-scale features and perform feature fusion to obtain a fused feature representation. The fusion feature representation is reconstructed to obtain the fused luminance channel image; The luminance channel image is combined with the Cb and Cr channels of the original medical source image to obtain a fused medical image.

[0010] Furthermore, the multi-scale image fusion network includes an upper branch, a middle branch, and a lower branch. The upper branch contains one adaptive convolutional module and three Transformer modules, the middle branch contains two adaptive convolutional modules and three Transformer modules, and the lower branch contains three adaptive convolutional modules and three Transformer modules. The number of adaptive convolutional modules varies in different scale branches to form feature extraction paths of different depths, thereby achieving multi-scale information complementarity.

[0011] Furthermore, the adaptive convolution module includes an adaptive convolution layer, a batch normalization layer, and a nonlinear activation layer. The adaptive convolution operation introduces a contextual information modeling mechanism on the basis of ordinary convolution to compensate for the shortcomings of ordinary convolution that only focuses on local information, thereby enhancing the ability to extract global semantic features in medical images.

[0012] Further, the step of inputting the original medical source image into a multi-scale image fusion network, extracting multi-scale features, and performing feature fusion to obtain a fused feature representation includes: The original medical source image is enhanced by an adaptive convolution module. By combining local convolution features with global contextual information, feature representations with global semantic awareness are extracted. The features processed by the adaptive convolution module are input into the Transformer module, which models the features through a multi-head self-attention mechanism and a feedforward network to capture long-range dependencies and global structural information in medical images.

[0013] Furthermore, the feature enhancement processing of the original medical source image based on the adaptive convolution module includes: performing forward calculation through the adaptive convolution module, as shown in the following formula: ; in, For input features, For adaptive convolution operations, For batch normalization operations, It is a non-linear activation function. The AC operation is used to combine local convolutional features with global contextual information to enhance the extraction of global semantic information.

[0014] Furthermore, the Transformer module includes layer normalization, multi-head self-attention mechanism and multilayer perceptron, and adopts residual connection structure. Through the Transformer module, long-distance dependencies in medical image features can be modeled, improving the ability of the fusion result to preserve structural information.

[0015] Furthermore, the step of inputting the features processed by the adaptive convolution module into the Transformer module includes: performing forward computation through the Transformer module, as shown in the following formula: ; ; in, For TM input features, This is the output of the first residual summation. For the final output features, For layer normalization operation, For multi-head self-attention operation, This is for multilayer perceptron operation.

[0016] Further, the reconstruction of the fused feature representation to obtain the fused luminance channel image specifically involves: The output features from Transformer modules at different scales are fused. The output features from each scale branch are added element-wise and then input into the output module. After processing with adaptive convolution and nonlinear activation functions, a fused brightness channel image is generated. The calculation process is as follows: ; in, , and These represent the Transformer output features from the upper, middle, and lower scale branches, respectively. This represents the fused luminance channel image.

[0017] Furthermore, the method includes: constructing a total loss function for network training, optimizing the multi-scale image fusion network without supervision from real fused images, and performing end-to-end training of the multi-scale image fusion network by minimizing the total loss function, wherein the total loss function consists of a content loss function and a structure loss function.

[0018] Furthermore, the network training employs a total loss function. The formula is as follows: ; in, This represents content loss, used to constrain the information integrity of the fused image; The structural loss is used to constrain the structural and texture information of the fused image. This is the balance coefficient.

[0019] Furthermore, the content loss Including regional mutual information constraints and pixel intensity constraints The formula is as follows: ; ; ; in, It represents regional mutual information, used to measure the statistical correlation between two images at the local region level; The brightness channel representing the fluorescent protein image; The brightness channel represents the phase difference microscopic image; Represents the brightness channel of the merged image; This is the first weighting coefficient, used to adjust the contribution ratio of the mutual information term in the GFP image region; This is the second weighting coefficient, used to adjust the contribution ratio of the mutual information term in the phase difference image region; The pixel value in the i-th row and j-th column of the merged image; Let be the pixel value of the source image A in the i-th row and j-th column; Let H be the pixel value of the source image B in the i-th row and j-th column; H and W are the height and width of the image.

[0020] Furthermore, the structural loss Including structural similarity constraints and gradient constraints The formula is as follows: ; ; ; in, It is a structural similarity index used to measure the degree of similarity between two images in terms of brightness, contrast, and structural information; These are the structural loss weighting coefficients, used to adjust the importance of preserving the structure of the phase difference image; For the Laplacian gradient operator, It is an L1 norm.

[0021] The beneficial effects of this invention are: 1. This invention constructs a multi-scale image fusion network structure to model the features of fluorescent protein microscopy images and phase difference microscopy images at different spatial scales in parallel, and combines an adaptive convolution module to achieve the collaborative extraction of local detail information and global semantic information, thereby significantly enhancing the detail expression ability and information integrity of the fused image; 2. Based on multi-scale feature modeling, this invention introduces a Transformer module to effectively capture the long-range dependencies between microscopic image features, improve the continuity and consistency of the fused image at the overall structural level, and ensure that the fusion result retains the structural features of the phase difference image while fully preserving the functional information of the fluorescent protein image. 3. This invention adopts a joint loss constraint method that combines content loss function and structure loss function, and avoids the use of adversarial training mechanism. While ensuring the stability and repeatability of the model training process, it achieves dual constraints on the fused image at the pixel level and the structural level, thereby obtaining a stable and robust microscopic image fusion result. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall process of the method of the present invention; Figure 2 A schematic diagram illustrating the structure of an adaptive convolutional module and its differences from that of a regular convolutional module. Figure 3 This is a schematic diagram of the Transformer module structure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0025] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and specific examples. The following drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] like Figure 1As shown, this embodiment of the invention provides a medical image fusion method based on multi-scale adaptive convolution and Transformer, including the following steps: Step 1: Obtain the original medical source images to be fused, including grayscale medical images and color medical images. When the source image is a color medical image, convert it from the RGB color space to the YCbCr color space, extract the Y channel in the YCbCr color space as the brightness information input to the network, and use the Cb and Cr channels for color reconstruction.

[0027] Step 2: Construct a multi-scale image fusion network, which includes several parallel branches of different scales. Each branch consists of at least one adaptive convolution module (ACM) and several Transformer modules (TM) connected in series.

[0028] Step 3: Input the converted original medical source image into a multi-scale image fusion network, extract multi-scale features and perform feature fusion to obtain a fused feature representation.

[0029] Step 4: Reconstruct the fused feature representation to obtain the fused luminance channel image.

[0030] Step 5: Combine the brightness channel image with the Cb and Cr channels of the original medical source image to obtain a fused medical image.

[0031] As a preferred embodiment, the conversion relationship from RGB color space to YCbCr color space in step 1 is as follows: ; In this embodiment, the Y channel represents luminance information, and the Cb and Cr channels represent chromaticity information. Only the Y channel is used as the network input for subsequent medical image fusion processing, while the Cb and Cr channels are used for color reconstruction of the final fusion result.

[0032] As a preferred embodiment, the multi-scale image fusion network constructed in step 2 specifically includes an upper branch, a middle branch, and a lower branch. The upper branch contains one adaptive convolutional module and three Transformer modules, the middle branch contains two adaptive convolutional modules and three Transformer modules, and the lower branch contains three adaptive convolutional modules and three Transformer modules.

[0033] By using branch structures of different depths to achieve multi-scale feature complementarity, the expressive power of information at different scales in medical images can be enhanced.

[0034] In a preferred embodiment, step 3 includes the following sub-steps: Step 3.1: Extract global semantic features based on adaptive convolution module The original medical source image obtained in step 1 is input into the adaptive convolution module for feature extraction. The adaptive convolution module sequentially includes an adaptive convolution layer, a batch normalization layer, and a non-linear activation layer, as shown below. Figure 2 As shown, its forward computation process is as follows: ; Where In represents the input features, Out represents the output features, and AC represents the adaptive convolution operation. The adaptive convolution introduces a contextual information modeling mechanism on top of ordinary convolution to compensate for the shortcomings of ordinary convolution, which only focuses on local information, thereby enhancing the ability to extract global semantic features from medical images.

[0035] Step 3.2: Model long-range dependencies based on the Transformer module The features processed by the adaptive convolution module are input into the Transformer module. The Transformer module includes layer normalization, a multi-head self-attention mechanism, and a multi-layer perceptron, and employs a residual connection structure, such as... Figure 3 As shown, its forward computation process is as follows: The first residual calculation process is as follows: ; The second residual calculation process is as follows: ; Here, MSA represents multi-head self-attention mechanism, and MLP represents multilayer perceptron. For TM input features, This is the output of the first residual summation. This is the final output feature. The Transformer module models long-range dependencies in medical image features, improving the ability of the fusion result to preserve structural information.

[0036] In a preferred embodiment, step 4 specifically involves: The output features from Transformer modules at different scales are fused. The output features from each scale branch are summed element-wise and then input into the output module. After adaptive convolution and nonlinear activation function processing, the fused brightness channel image is generated. The calculation process is as follows: ; in, , and These represent the Transformer output features from the upper, middle, and lower scale branches, respectively. This represents the fused luminance channel image.

[0037] In a preferred embodiment, step 5 specifically involves: The brightness channel image The images are then combined with the original Cb and Cr chromaticity channels retained in step 1 and converted back to the RGB color space to obtain the final fused medical image. The conversion relationship is as follows: .

[0038] In a preferred embodiment, the method further includes step 6: constructing a loss function and training the network, specifically: Construct the total loss function for network training The network is optimized without supervision from real-world fused images. The total loss function consists of a content loss function and a structure loss function, and its expression is as follows: ; in, As a weighting factor, Content loss is used to constrain the information integrity of the fused image. The structural loss is used to constrain the structural and texture information of the fused image.

[0039] The content loss function includes a region mutual information constraint term and a pixel intensity constraint term, and its calculation process is as follows: ; ; ; in, It represents regional mutual information, used to measure the statistical correlation between two images at the local region level; The brightness channel representing the fluorescent protein image; The brightness channel represents the phase difference microscopic image; Represents the brightness channel of the merged image; This is the first weighting coefficient, used to adjust the contribution ratio of the mutual information term in the GFP image region; This is the second weighting coefficient, used to adjust the contribution ratio of the mutual information term in the phase difference image region; The pixel value in the i-th row and j-th column of the merged image; Let be the pixel value of the source image A in the i-th row and j-th column; Let H be the pixel value of the source image B in the i-th row and j-th column; H and W are the height and width of the image.

[0040] The structural loss function includes structural similarity constraint terms and gradient constraint terms, and its calculation process is as follows: ; ; ; in, It is a structural similarity index used to measure the degree of similarity between two images in terms of brightness, contrast, and structural information; These are the structural loss weighting coefficients, used to adjust the importance of preserving the structure of the phase difference image; For the Laplacian gradient operator, It is an L1 norm.

[0041] By minimizing the above total loss function, the network is trained end-to-end to achieve effective fusion of structural and functional information in medical images.

[0042] The final result of this invention is that by taking two images as input, a fused image of the two images can be obtained, as shown in the image below. Figure 1 As shown.

[0043] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A medical image fusion method based on multi-scale adaptive convolution and Transformer, characterized in that, Includes the following steps: The original medical source images to be fused are obtained, including grayscale medical images and color medical images. When the source image is a color medical image, it is converted from the RGB color space to the YCbCr color space. The Y channel in the YCbCr color space is extracted as the brightness information input to the network, and the Cb and Cr channels are used for color reconstruction. A multi-scale image fusion network is constructed, which includes several parallel branches of different scales. Each branch consists of at least one adaptive convolutional module and several Transformer modules connected in series. The original medical source image is input into a multi-scale image fusion network to extract multi-scale features and perform feature fusion to obtain a fused feature representation. The fusion feature representation is reconstructed to obtain the fused luminance channel image; The luminance channel image is combined with the Cb and Cr channels of the original medical source image to obtain a fused medical image.

2. The method according to claim 1, characterized in that, The multi-scale image fusion network includes an upper branch, a middle branch, and a lower branch. The upper branch contains one adaptive convolutional module and three Transformer modules, the middle branch contains two adaptive convolutional modules and three Transformer modules, and the lower branch contains three adaptive convolutional modules and three Transformer modules.

3. The method according to claim 1, characterized in that, The step of inputting the original medical source image into a multi-scale image fusion network, extracting multi-scale features and performing feature fusion to obtain a fused feature representation includes: The original medical source image is enhanced by an adaptive convolution module. By combining local convolution features with global contextual information, feature representations with global semantic awareness are extracted. The features processed by the adaptive convolution module are input into the Transformer module, which models the features through a multi-head self-attention mechanism and a feedforward network to capture long-range dependencies and global structural information in medical images.

4. The method according to claim 3, characterized in that, The feature enhancement processing of the original medical source image based on the adaptive convolution module includes: performing forward calculation through the adaptive convolution module, as shown in the following formula: ； in, For input features, For adaptive convolution operations, For batch normalization operations, It is a non-linear activation function. The AC operation is used to combine local convolutional features with global contextual information to enhance the extraction of global semantic information.

5. The method according to claim 3, characterized in that, The step of inputting the features processed by the adaptive convolution module into the Transformer module includes: performing forward computation through the Transformer module, as shown in the following formula: ； ； in, For TM input features, This is the output of the first residual summation. For the final output features, For layer normalization operation, For multi-head self-attention operation, This is for multilayer perceptron operation.

6. The method according to claim 1, characterized in that, The process of reconstructing the fused feature representation to obtain the fused brightness channel image is as follows: The output features from Transformer modules at different scales are fused. The output features from each scale branch are added element-wise and then input into the output module. After processing with adaptive convolution and nonlinear activation functions, a fused brightness channel image is generated. The calculation process is as follows: ； in, , and These represent the Transformer output features from the upper, middle, and lower scale branches, respectively. This represents the fused luminance channel image.

7. The method according to claim 1, characterized in that, The method includes: constructing a total loss function for network training, optimizing the multi-scale image fusion network without supervision from real fused images, and performing end-to-end training of the multi-scale image fusion network by minimizing the total loss function, wherein the total loss function consists of a content loss function and a structure loss function.

8. The method according to claim 7, characterized in that, The network training uses a total loss function. The formula is as follows: ； in, This represents content loss, used to constrain the information integrity of the fused image; The structural loss is used to constrain the structural and texture information of the fused image. This is the balance coefficient.

9. The method according to claim 8, characterized in that, The content loss Including regional mutual information constraints and pixel intensity constraints The formula is as follows: ； ； ； in, It represents regional mutual information, used to measure the statistical correlation between two images at the local region level; The brightness channel representing the fluorescent protein image; The brightness channel represents the phase difference microscopic image; Represents the brightness channel of the merged image; This is the first weighting coefficient, used to adjust the contribution ratio of the mutual information term in the GFP image region; This is the second weighting coefficient, used to adjust the contribution ratio of the mutual information term in the phase difference image region; The pixel value in the i-th row and j-th column of the merged image; Let be the pixel value of the source image A in the i-th row and j-th column; Let H be the pixel value of the source image B in the i-th row and j-th column; H and W are the height and width of the image.

10. The method according to claim 8, characterized in that, The structural loss Including structural similarity constraints and gradient constraints The formula is as follows: ； ； ； in, It is a structural similarity index used to measure the degree of similarity between two images in terms of brightness, contrast, and structural information; These are the structural loss weighting coefficients, used to adjust the importance of preserving the structure of the phase difference image; For the Laplacian gradient operator, It is an L1 norm.

Images