Multimodal medical image fusion method and system based on functional-anatomical transformer

Abstract

The application discloses a multi-modal medical image fusion method and system based on function-anatomy Transformer. First, a multi-modal medical fusion model is constructed: the SPECT image is subjected to RGB-YUV color space conversion to obtain three components Y, U and V; the Y component is input into a function multi-scale branch to extract multi-scale function features; meanwhile, the MRI is input into an anatomy multi-scale branch to extract multi-scale anatomy features; a function-guided Transformer module and an anatomy-guided Transformer module are designed to interact the multi-scale function features and the multi-scale anatomy features; the extracted multi-scale function features and the multi-scale anatomy features are integrated and reconstructed to obtain the Y component of the fusion image; pixel loss and gradient loss are constructed to constrain the fusion image to have similar pixel information as the source image and guide the fusion image to have similar gradient distribution as the source image; the pixel loss and the gradient loss are combined to jointly optimize the constructed multi-modal medical image fusion model; in the test stage, the optimized multi-modal medical image fusion model is used to obtain a fusion result.

Description

Technical Field

[0001] This invention belongs to the fields of pattern recognition and artificial intelligence, specifically relating to a multimodal medical image fusion method and system based on functional-anatomical Transformer. Background Technology

[0002] Image fusion is a long-standing and challenging topic in computer vision, with wide applications in smart healthcare, assisted diagnosis, object detection, and recognition. Due to differences and limitations in imaging techniques, single-modal images struggle to display comprehensive information. Specifically, SPECT images provide information related to metabolism, allowing observation of abnormal lesions, but soft tissue structures are unclear; MRI images, on the other hand, reveal rich soft tissue information. Multimodal medical image fusion integrates complementary information from different medical modalities using computer technology, discarding redundant information to generate a rich synthetic image that simultaneously displays significant lesions and clear soft tissue structures. Multimodal medical images have received increasing attention due to their value in medical imaging diagnosis. However, most existing deep fusion models do not fully consider the distinct characteristics of different modalities, leading to information loss. Summary of the Invention

[0003] The purpose of this invention is to solve the problems of large differences in multimodal medical images, especially the difficulty in information extraction and the distinct modal features. To this end, this invention provides a multimodal medical image fusion method and computing system based on functional-anatomical Transformer.

[0004] To achieve the aforementioned objective, the present invention proposes a multimodal medical image fusion method based on functional-anatomical Transformer, comprising the following steps:

[0005] A multimodal medical image fusion method based on functional-anatomical Transformer is characterized by the following steps:

[0006] Step 1: Perform RGB-YUV color space conversion on the single-photon emission computed tomography (SPECT) image to obtain three components: Y, U, and V. Y represents luminance, U represents hue, and V represents color saturation.

[0007] Step 2: Based on convolutional neural networks and Transformers, construct a multimodal medical image fusion model. Its input is the Y component of SPECT images and MRI, and its output is the Y component of the fused image.

[0008] Step 3: Construct a loss function to optimize the fusion model;

[0009] Step 4: Use the trained fusion model to extract complementary features from the source images, discard redundant features, and then obtain the fusion result through YUV-RGB color space conversion.

[0010] Furthermore, the fusion model includes a multi-scale feature extraction module and a feature reconstruction module. The multi-scale feature extraction module includes two branches: an anatomical multi-scale branch consisting of N feature extraction modules and N anatomically guided Transformer modules, used to extract multi-scale functional features, i.e., soft tissue structure information, from MRI images; and a functional multi-scale branch consisting of N feature extraction modules and N functionally guided Transformer modules, used to extract multi-scale anatomical features, i.e., metabolic features, from SPECT images.

[0011] The extracted multi-scale functional features and multi-scale anatomical features are integrated and reconstructed to obtain integrated multi-scale functional features and multi-scale anatomical features.

[0012] The feature reconstruction module includes an anatomy-guided Transformer module and a function-guided Transformer module.

[0013] Multi-scale functional features and multi-scale anatomical features are integrated by interactively using the function-guided Transformer module and the anatomy-guided Transformer module, and then the images are stitched together and passed through a convolution module to output a fused image.

[0014] Furthermore, each of the N feature extraction modules consists of a convolutional layer, a batch normalization layer, and a nonlinear ReLU layer.

[0015] Furthermore, the specific processing procedure of the function-guided Transformer module is as follows:

[0016] The function-guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. The integrated multi-scale functional features are input into the channel attention modules to extract important functional information. The extracted important functional information and the integrated multi-scale anatomical features are then concatenated along the channel dimension. The concatenated features are then sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a ReLU nonlinear layer to integrate the features and reduce dimensionality. The integrated features are then input into a channel attention module to extract important complementary features. Finally, the features are sequentially input into two Transformer modules to extract global features.

[0017] Furthermore, the specific processing procedure of the dissection-guided Transformer module is as follows:

[0018] The anatomical-guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. The integrated multi-scale anatomical features are input into the channel attention modules to extract important anatomical information. The extracted important anatomical information and the integrated multi-scale functional features are concatenated along the channel dimension. The concatenated features are then sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a ReLU nonlinear layer to integrate the features and reduce dimensionality. The integrated features are then input into a channel attention module to extract important complementary features. Finally, they are sequentially input into two Transformer modules to extract global features.

[0019] Furthermore, the specific processing procedure of the channel attention module is as follows:

[0020] The input features are sequentially fed into a global average pooling layer, a 1×1 convolutional layer, a non-linear ReLU layer, a 1×1 convolutional layer, and a non-linear Sigmoid layer to extract features; the extracted features are multiplied by the input features to obtain the output features.

[0021] Furthermore, the specific processing procedure of the Transformer module is as follows:

[0022] The Transformer module consists of two skip connections in sequence; the first skip connection consists of a layer normalization and multi-head self-attention; the second skip connection consists of a layer normalization and multi-layer perception.

[0023] Furthermore, the specific implementation method for obtaining the integrated multi-scale functional features and multi-scale anatomical features is as follows:

[0024] The extracted multi-scale anatomical features are concatenated along the channel dimension and then input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear ReLU layer to integrate the features and reduce the dimensionality.

[0025] The extracted multi-scale functional features are concatenated along the channel dimension and then input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear ReLU layer to integrate the features and reduce the dimensionality.

[0026] The convolution module consists of a 1×1 convolutional layer and a non-linear Sigmoid layer.

[0027] Furthermore, the loss function includes pixel loss and gradient loss;

[0028] Pixel loss L p Represented as:

[0029]

[0030] in, and These are respectively used to constrain the fused image and the functional image to exhibit similar pixel information, and the fused image and the structural image to exhibit similar pixel information. The charbonnier penalty function is expressed as:

[0031]

[0032] Where y is fun or ana, ε is the penalty coefficient, and I f and I z These are the fused image and the source image, respectively. The source image refers to the Y component of the SPECT image and the MRI image.

[0033] Gradient loss L tv Represented as:

[0034]

[0035] Among them, I f To merge images, It refers to the Y component of a SPECT image, I mri This refers to MRI images, where TV(·) is the total variation, represented as:

[0036]

[0037] Where (i,j) represents the position of image Z, β is a regularization term, is a constant, Z represents the fused image or the source image, and x ij This represents the pixel value located at (i,j);

[0038] The final loss function is expressed as:

[0039] L = L p +L tv .

[0040] This invention also provides a multimodal medical image fusion system based on functional-anatomical Transformer, comprising the following units:

[0041] The color space conversion unit performs RGB-YUV color space conversion on the single-photon emission computed tomography (SPECT) image to obtain three components: Y, U, and V. Y represents luminance, U represents hue, and V represents color saturation.

[0042] The fusion model building unit, based on convolutional neural networks and Transformer, constructs a multimodal medical image fusion model. Its input is the Y component of SPECT images and MRI, and its output is the Y component of the fused image.

[0043] The training optimization unit is used to construct the loss function and optimize the fusion model.

[0044] The fusion result output unit is used to extract complementary features from the source image using the trained fusion model, discard redundant features, and then obtain the fusion result through YUV-RGB color space conversion.

[0045] The beneficial effects of this invention are as follows: By constructing functional multi-scale branches and anatomical multi-scale branches, this invention fully extracts information within multiple modalities. Through the design of function-guided and anatomically-guided Transformer modules, it fully interacts and integrates multi-scale features between modalities. This invention fully integrates the functional information of SPECT images and the anatomical information of MRI to generate information-rich fused images that simultaneously contain metabolic information and soft tissue structure. Furthermore, by constructing pixel loss and gradient loss, it constrains the fusion model to generate information-rich fused images, thereby improving the performance of multimodal medical image fusion. Attached Figure Description

[0046] Figure 1 This is a flowchart of an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the structural framework of an embodiment of the present invention.

[0048] Figure 3-5 These are three sets of source images and their fusion results generated under different fusion methods in the embodiments of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0050] Figure 1 This is a flowchart of a multimodal medical image fusion method based on functional-anatomical Transformer according to an embodiment of the present invention. The following is an example... Figure 1 To illustrate some specific implementation processes of the present invention, we will use examples, such as... Figure 1 As shown, the multimodal medical image method based on functional-anatomical Transformer includes the following steps:

[0051] Step S1: Perform RGB-YUV color space conversion on the SPECT image to obtain three components: Y, U, and V.

[0052] Among them, color space conversion is an existing technology, Y represents brightness, U represents hue, and V represents color saturation; steps S2 and S3 have no sequential order.

[0053] Step S2: Construct a multimodal medical image fusion model based on convolutional neural networks and Transformers. This model consists of a multi-scale feature extraction module and a feature reconstruction module. The multi-scale feature extraction module includes two branches: an anatomical multi-scale branch composed of N feature extraction modules and N anatomically guided Transformer modules, used to extract soft tissue structure information from MRI images; and a functional multi-scale branch composed of N feature extraction modules and N functionally guided Transformer modules, used to extract metabolic features from SPECT images (Single-Photon Emission Computed Tomography, SPECT). The feature reconstruction module includes convolutional layers, anatomically guided Transformer modules, and functionally guided Transformer modules.

[0054] Further, step S2 includes the following steps:

[0055] Step S21: Based on convolutional neural networks and Transformers, design a functional multi-scale branch and an anatomical multi-scale branch to extract functional information from SPECT images and soft tissue structures from MRI images. The anatomical multi-scale branch consists of N feature extraction modules and N anatomically guided Transformer modules; the functional multi-scale branch consists of N feature extraction modules and N function-guided Transformer modules.

[0056] Step S22: Design N feature extraction modules to extract multi-scale information; in this embodiment, N is 3. The first feature extraction module consists of a 3×3 convolutional layer, a batch normalization layer, and a non-linear layer (ReLU), with 1 input and 16 output channels. The second feature extraction module consists of a 5×5 convolutional layer, a batch normalization layer, and a non-linear layer (ReLU), with 16 input and 64 output channels. The third feature extraction module consists of a 7×7 convolutional layer, a batch normalization layer, and a non-linear layer (ReLU), with 64 input and 256 output channels.

[0057] Step S3: Input the Y component of the SPECT image into the functional multi-scale branch and extract the multi-scale functional features;

[0058] In this embodiment, only the Y component is input into the network and participates in training, while the U and V components do not participate in network training and are only used for color conversion.

[0059] Furthermore, in step S3, N feature extraction modules and N function-guided Transformer modules are designed to extract multi-scale functional features;

[0060] Step S4: Input MRI (Magnetic Resonance Imaging) into the anatomical multi-scale branch to extract multi-scale anatomical features;

[0061] Furthermore, in step S4, N feature extraction modules and N anatomically guided Transformer modules are designed to dissect multi-scale functional features;

[0062] Step S5: Integrate and reconstruct the extracted multi-scale functional features and multi-scale anatomical features to obtain the integrated multi-scale functional features and multi-scale anatomical features.

[0063] Further, step S5 includes the following steps:

[0064] Step S51: The extracted multi-scale anatomical features are concatenated along the channel dimension and input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a nonlinear layer (ReLU) to integrate the features and reduce the dimensionality (the number of input and output channels are 336 and 16, respectively).

[0065] Step S52: The extracted multi-scale functional features are concatenated along the channel dimension and input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a nonlinear layer (ReLU) to integrate the features and reduce the dimensionality (the number of input and output channels are 336 and 16, respectively).

[0066] Step S53: Input the integrated multi-scale anatomical features and multi-scale functional features into the anatomically guided Transformer module;

[0067] Step S54: Input the integrated multi-scale anatomical features and multi-scale functional features into the function-guided Transformer module;

[0068] Step S6: Design multi-scale functional features and multi-scale anatomical features after interactive integration of the function-guided Transformer module and the anatomy-guided Transformer module.

[0069] Further, step S6 includes the following steps:

[0070] Step S61: Design a function-guided Transformer module to interactively extract metabolic information. The function-guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. The integrated multi-scale functional features are input into the channel attention modules to extract important functional information; the extracted important functional information and the integrated multi-scale anatomical features are concatenated along the channel dimension; the concatenated features are sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear layer (ReLU) to integrate the features and reduce the dimensionality (the number of input and output channels are 32 and 16, respectively); the integrated features are input into a channel attention module to extract important complementary features; and then sequentially input into two Transformer modules to extract global features.

[0071] Step S62: Design an anatomically guided Transformer module to interactively extract soft tissue structural information. The anatomically guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. The integrated multi-scale anatomical features are input into the channel attention modules to extract important anatomical information; the extracted important anatomical information and the integrated multi-scale functional features are concatenated along the channel dimension; the concatenated features are sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a nonlinear layer (ReLU) to integrate the features and reduce the dimensionality (the number of input and output channels are 32 and 16, respectively); the integrated features are input into a channel attention module to extract important complementary features; and then sequentially input into two Transformer modules to extract global features.

[0072] Step S63: Design a channel attention module to extract important features. The channel attention module sequentially inputs the input features into a global average pooling layer, a 1×1 convolutional layer, a non-linear layer (ReLU), a 1×1 convolutional layer, and a non-linear layer (Sigmoid) to extract features; the extracted features are multiplied with the input features to obtain the output features.

[0073] Step S64: The features extracted by the anatomy-guided Transformer module and the function-guided Transformer module are concatenated in the channel dimension and input into a convolutional layer consisting of a 1×1 convolutional layer and a non-linear layer (Sigmoid) to generate the Y component of the fused image.

[0074] Step S7: Construct pixel loss and constrain the fused image to have similar pixel information to the source image;

[0075] Furthermore, in step S7, the fused image is constrained to exhibit similar pixel information to the source image, and a pixel loss L is constructed. p , is represented as:

[0076]

[0077] in, and These are respectively used to constrain the fused image and the functional image to exhibit similar pixel information, and the fused image and the structural image to exhibit similar pixel information. The charbonnier penalty function is expressed as:

[0078]

[0079] Where y is fun or ana, ε is the penalty coefficient, and I f and I z These are the fused image and the source image, respectively. The source image refers to the Y component of the SPECT image and the MRI image.

[0080] Step S8: Construct gradient loss to guide the fused image to have a similar gradient distribution to the source image;

[0081] Furthermore, in step S8, the fused image is guided to have a similar gradient distribution to the source image, and a gradient loss is constructed:

[0082]

[0083] Among them, I f To merge images, It refers to the Y component of a SPECT image, I mti This refers to MRI images, where TV(·) is the total variation, represented as:

[0084]

[0085] Where (i,j) represents the position of image Z, β is a regularization term, is a constant, Z represents the fused image or the source image, and x ij This represents the pixel value located at (i,j).

[0086] Step S9: Combine pixel-level loss and gradient loss to jointly optimize the constructed multimodal medical image fusion model.

[0087] Furthermore, in step S8, the final constraint function of the multimodal medical image fusion model jointly optimized by pixel-level loss and structural loss is expressed as:

[0088] L = L p +L tv

[0089] Step S10: Based on the optimized multimodal medical image fusion model, extract complementary features from the source images, discard redundant features, and obtain the fusion result through YUV-RGB color space conversion.

[0090] On the other hand, the present invention also provides a multimodal medical image fusion system based on functional-anatomical Transformer, comprising the following units:

[0091] This invention also provides a multimodal medical image fusion system based on functional-anatomical Transformer, comprising the following units:

[0092] The color space conversion unit performs RGB-YUV color space conversion on the single-photon emission computed tomography (SPECT) image to obtain three components: Y, U, and V. Y represents luminance, U represents hue, and V represents color saturation.

[0093] The fusion model building unit, based on convolutional neural networks and Transformer, constructs a multimodal medical image fusion model. Its input is the Y component of SPECT images and MRI, and its output is the Y component of the fused image.

[0094] The training optimization unit is used to construct the loss function and optimize the fusion model.

[0095] The fusion result output unit is used to extract complementary features from the source image using the trained fusion model, discard redundant features, and then obtain the fusion result through YUV-RGB color space conversion.

[0096] The specific implementation methods of each unit are the same as those of each step, and will not be described in this invention.

[0097] The present invention also provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal medical image fusion method.

[0098] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described multimodal medical image fusion method.

[0099] Finally, as an example, this application compares with existing multimodal medical image fusion methods. Specifically, the comparison methods include four representative methods: CVT, SR, LLF, and NSST; four state-of-the-art convolutional neural network-based methods: CNN, U2Fusion, DDcGA, and EMFusion; and four state-of-the-art Transformer-based methods: PPT Fusion, SwinFuse, SwinFusion, and MATR. To further visually demonstrate the objective performance of different comparison methods, seven commonly used objective evaluation metrics were employed, including Q... MI Q G Q P Q Y Q CB VIF and MI abf Table 1 shows the objective performance of different fusion methods, with the best and second-best values ​​indicated by bold and underline, respectively. It can be seen that the method in this application achieves satisfactory objective performance.

[0100] Table 1 Objective performance of different fusion methods

[0101]

[0102] Subjective comparison results reference Figure 3-5 ; Figure 3-5 It consists of three sets of source images and their fusion results generated under different fusion methods.

[0103] Depend on Figure 3-5 It can be seen that the method of this application can simultaneously preserve metabolic information in SPECT and soft tissue details in MRI images, and retains more complementary information from source images than other multimodal medical image fusion methods.

[0104] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A multimodal medical image fusion method based on functional-anatomical Transformer, characterized in that, Includes the following steps: Step 1: Perform RGB-YUV color space conversion on the single-photon emission computed tomography (SPECT) image to obtain three components: Y, U, and V. Y represents luminance, U represents hue, and V represents color saturation. Step 2: Based on convolutional neural networks and Transformers, construct a multimodal medical image fusion model. Its input is the Y component of SPECT images and MRI, and its output is the Y component of the fused image. The multimodal medical image fusion model includes a multi-scale feature extraction module and a feature reconstruction module. The multi-scale feature extraction module includes two branches: an anatomical multi-scale branch consisting of N feature extraction modules and N anatomically guided Transformer modules, used to extract multi-scale functional features, i.e., soft tissue structure information, from MRI images; and a functional multi-scale branch consisting of N feature extraction modules and N functionally guided Transformer modules, used to extract multi-scale anatomical features, i.e., metabolic features, from SPECT images. The extracted multi-scale functional features and multi-scale anatomical features are integrated and reconstructed to obtain integrated multi-scale functional features and multi-scale anatomical features. The feature reconstruction module includes an anatomy-guided Transformer module and a function-guided Transformer module. Multi-scale functional features and multi-scale anatomical features are integrated by interactively using the function-guided Transformer module and the anatomy-guided Transformer module, and then the images are stitched together and passed through a convolution module to output a fused image. Step 3: Construct a loss function to optimize the fusion model; The loss function includes pixel loss and gradient loss; Pixel loss Represented as: in, and These are respectively used to constrain the fused image and the functional image to exhibit similar pixel information, and the fused image and the structural image to exhibit similar pixel information. The charbonnier penalty function is expressed as: in, y For fun or ana, The penalty coefficient is... and These are the fused image and the source image, respectively. The source image refers to the Y component of the SPECT image and the MRI image. Gradient loss Represented as: in, To merge images, This refers to the Y component of a SPECT image. Refers to MRI images. The total variation is expressed as: Where (i,j) represents the position of image Z. Here, is the regularization term, is a constant, and Z represents the fused image or the source image. The pixel value at (i,j); The final loss function is expressed as: Step 4: Use the trained fusion model to extract complementary features from the source images, discard redundant features, and then obtain the fusion result through YUV-RGB color space conversion.

2. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 1, characterized in that: Each of the N feature extraction modules consists of a convolutional layer, a batch normalization layer, and a nonlinear ReLU layer.

3. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 1, characterized in that: The specific processing procedure of the Transformer module guided by the aforementioned function is as follows: The function-guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. It inputs the integrated multi-scale functional features into the channel attention modules to extract important functional information; and then concatenates the extracted important functional information and the integrated multi-scale anatomical features along the channel dimension. The concatenated features are sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear ReLU layer to integrate the features and reduce the dimensionality; the integrated features are then input into a channel attention module to extract important complementary features; and finally, they are sequentially input into two Transformer modules to extract global features.

4. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 1, characterized in that: The specific processing procedure of the dissection-guided Transformer module is as follows: The anatomical-guided Transformer module consists of two channel attention modules, two Transformer modules, and a convolutional layer. The integrated multi-scale anatomical features are input into the channel attention modules to extract important anatomical information. The extracted important anatomical information and the integrated multi-scale functional features are concatenated along the channel dimension. The concatenated features are then sequentially input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a ReLU nonlinear layer to integrate the features and reduce dimensionality. The integrated features are then input into a channel attention module to extract important complementary features. Finally, they are sequentially input into two Transformer modules to extract global features.

5. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 3 or 4, characterized in that: The specific processing procedure of the channel attention module is as follows: The input features are sequentially fed into a global average pooling layer, a 1×1 convolutional layer, a non-linear ReLU layer, a 1×1 convolutional layer, and a non-linear Sigmoid layer to extract features; the extracted features are multiplied by the input features to obtain the output features.

6. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 3 or 4, characterized in that: The specific processing procedure of the Transformer module is as follows: The Transformer module consists of two skip connections in sequence; the first skip connection consists of a layer normalization and multi-head self-attention; the second skip connection consists of a layer normalization and multi-layer perception.

7. The multimodal medical image fusion method based on functional-anatomical Transformer as described in claim 1, characterized in that: The specific implementation method for obtaining the integrated multi-scale functional features and multi-scale anatomical features is as follows: The extracted multi-scale anatomical features are concatenated along the channel dimension and then input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear ReLU layer to integrate the features and reduce the dimensionality. The extracted multi-scale functional features are concatenated along the channel dimension and then input into a convolutional layer consisting of a 1×1 convolutional layer, a batch normalization layer, and a non-linear ReLU layer to integrate the features and reduce the dimensionality. The convolution module consists of a 1×1 convolutional layer and a non-linear Sigmoid layer.

8. A multimodal medical image fusion system based on functional-anatomical Transformer, characterized in that, Includes the following units: The color space conversion unit performs RGB-YUV color space conversion on the single-photon emission computed tomography (SPECT) image to obtain three components: Y, U, and V. Y represents luminance, U represents hue, and V represents color saturation. The fusion model building unit, based on convolutional neural networks and Transformer, constructs a multimodal medical image fusion model. Its input is the Y component of SPECT images and MRI, and its output is the Y component of the fused image. The multimodal medical image fusion model includes a multi-scale feature extraction module and a feature reconstruction module. The multi-scale feature extraction module includes two branches: an anatomical multi-scale branch consisting of N feature extraction modules and N anatomically guided Transformer modules, used to extract multi-scale functional features, i.e., soft tissue structure information, from MRI images; and a functional multi-scale branch consisting of N feature extraction modules and N functionally guided Transformer modules, used to extract multi-scale anatomical features, i.e., metabolic features, from SPECT images. The extracted multi-scale functional features and multi-scale anatomical features are integrated and reconstructed to obtain integrated multi-scale functional features and multi-scale anatomical features. The feature reconstruction module includes an anatomy-guided Transformer module and a function-guided Transformer module. Multi-scale functional features and multi-scale anatomical features are integrated by interactively using the function-guided Transformer module and the anatomy-guided Transformer module, and then the images are stitched together and passed through a convolution module to output a fused image. The training optimization unit is used to construct the loss function and optimize the fusion model; The loss function includes pixel loss and gradient loss; Pixel loss Represented as: in, and These are respectively used to constrain the fused image and the functional image to exhibit similar pixel information, and the fused image and the structural image to exhibit similar pixel information. The charbonnier penalty function is expressed as: in, y For fun or ana, The penalty coefficient is... and These are the fused image and the source image, respectively. The source image refers to the Y component of the SPECT image and the MRI image. Gradient loss Represented as: in, To merge images, This refers to the Y component of a SPECT image. Refers to MRI images. The total variation is expressed as: Where (i,j) represents the position of image Z. Here, is the regularization term, is a constant, and Z represents the fused image or the source image. The pixel value at (i,j); The final loss function is expressed as: The fusion result output unit is used to extract complementary features from the source image using the trained fusion model, discard redundant features, and then obtain the fusion result through YUV-RGB color space conversion.

Images