Three-modal medical image fusion method and system based on primitive reasoning network

Abstract

The application discloses a three-modal medical image fusion method and system based on a primitive reasoning network, and the method comprises the following steps: acquiring three-modal medical images and performing image preprocessing; performing energy calculation processing on the preprocessed three-modal medical images based on a multi-scale primitive reasoning attention network, and acquiring corresponding energy maps; combining the preprocessed three-modal medical images with the corresponding energy maps, and performing fusion processing through an energy ratio fusion strategy to output a preliminary fusion image; and performing discrimination processing on the preliminary fusion image based on a discriminator and a generator to construct an adversarial game relationship model and generate a fusion image containing more third-class source image information. The application fuses different and complementary information in multi-modal medical images into one image, helps doctors better analyze difficult and complicated diseases, and reduces misdiagnosis and surgical errors. The application can be widely applied to the field of image fusion technology.

Description

Technical Field

[0001] This invention relates to the field of image processing data fusion technology, and in particular to a three-modal medical image fusion method and system based on primitive inference networks. Background Technology

[0002] Different medical imaging modalities have been used in clinical disease diagnosis for many years. However, due to the differences in sensors, a single type of medical image cannot provide doctors with comprehensive diagnostic information. Multimodal medical image fusion technology can integrate useful information from different modalities of medical images into a single synthetic image. Specifically, multimodal medical images include, but are not limited to, CT, MRI, PET, and SPECT. MRI and CT are anatomical images, mainly used to display the internal anatomical structures and morphology of the human body. CT images primarily provide high-resolution and detailed information on bone and high-density tissue structures, while MRI is mainly used to generate high-resolution information on soft tissue structures within the human body. MRI includes MR-T1 and MR-T2; MR-T1 better reflects anatomical structures, while MR-T2 is mainly used to detect pathological changes in tissues. Functional medical images can provide metabolic information within the human body. PET and SPECT imaging, as functional techniques, play a crucial role in the medical field. PET contains information about tumor function and metabolic activity and is commonly used to diagnose tumors, neurological diseases, etc. However, functional medical images have a significant drawback: low resolution. In summary, medical image fusion can eliminate redundant information while complementing each other's strengths, generating a fused image that contains both anatomical structural and morphological information and functional metabolic activity information. Medical image fusion can provide a more comprehensive and reliable description of lesions, thus aiding biomedical research and clinical diagnosis. For example, CT / MR image fusion can assist in preoperative examination of temporal bone tumors, and MR / SPECT image fusion can locate epileptogenic foci. Multimodal medical image fusion is typically performed at the pixel level and can be broadly categorized into traditional methods and deep learning methods. Although traditional methods have achieved good fusion results, the pixel activity level measurement and weight allocation strategies are designed separately and are not strongly correlated with the fusion method, greatly limiting algorithm performance. Therefore, it is difficult for traditional methods to design an ideal pixel activity level measurement or weight allocation strategy that comprehensively considers all key issues. However, due to the excellent feature extraction and data representation capabilities of convolutional neural networks, many deep learning-based fusion methods have been proposed.

[0003] However, existing neural network frameworks still have some problems. First, the decomposition and reconstruction process of fusion methods based on autoencoder models is time-consuming and complex, and overfitting is prone to occur during training. Second, most image fusion methods based on convolutional neural networks rely on convolution operations, so their ability to extract global features is weak. Third, even though Transformer models have the ability to model long-distance contextual dependencies, they require more computational resources and time. Fourth, current medical image fusion methods require step-by-step operations to complete trimodal medical image fusion. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a trimodal medical image fusion method and system based on primitive reasoning networks. By fusing different and complementary information from original multimodal medical images into a single image, it can help doctors better analyze complex and difficult cases and reduce misdiagnosis and surgical errors.

[0005] The first technical solution adopted in this invention is: a three-modal medical image fusion method based on primitive inference networks, comprising the following steps:

[0006] A trimodal medical image is acquired and preprocessed to obtain a preprocessed trimodal medical image, wherein the trimodal medical image includes a first type of source image, a second type of source image, and a third type of source image;

[0007] Energy calculation is performed on preprocessed trimodal medical images using a multi-scale primitive inference attention network to obtain the corresponding energy maps;

[0008] The preprocessed trimodal medical images are combined with the corresponding energy maps, and the combination results are fused using an energy ratio fusion strategy to output a preliminary fused image.

[0009] An adversarial game relationship model based on a discriminator and a generator is constructed to perform preliminary discrimination processing on the fused image, generating a fused image containing more information from the third type of source image, and obtaining the final fused image.

[0010] Furthermore, the step of acquiring trimodal medical images and performing image preprocessing to obtain preprocessed trimodal medical images specifically includes:

[0011] Acquire trimodal medical images, which include a first type of source image, a second type of source image, and a third type of source image. The first type of source image includes MR-T2 images, the second type of source image includes PET images and SPECT images, and the third type of source image includes MR-Gad images, MR-T1 images, and CT images.

[0012] Convert the second type of source image into a YCRCB image to obtain the second type of source image in the Y channel;

[0013] By combining the first type of source image, the second type of source image under the Y channel, and the third type of source image, a preprocessed trimodal medical image is constructed.

[0014] Furthermore, the step of performing energy calculation processing on the preprocessed trimodal medical images based on a multi-scale primitive inference attention network to obtain the corresponding energy map specifically includes:

[0015] The preprocessed trimodal medical images are input into a multi-scale primitive relation inference attention network, which includes a first squeezing and excitation attention block, a multi-scale module with residual connections, a second multi-scale block, a squeezing and excitation inference attention module, and a convolutional layer.

[0016] The first squeezing and excitation attention block performs convolution processing on the preprocessed trimodal medical image and sends the convolutional feature map to convolution branches of different scales. The multi-scale module with residual connections is used to extract multi-scale features of the source image to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature maps to generate multiple feature maps with global and local features. The feature maps are then added element-wise with the input feature map of the multi-scale module with residual connections to obtain the first output result.

[0017] The first output result is convolved based on the second multi-scale block, and the feature map obtained by convolution is sent to convolution branches of different scales. The multi-scale module is used to extract multi-scale features of the feature map to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature map to generate multiple feature maps with global and local features.

[0018] All the obtained feature maps are input into the final convolutional layer of the multi-scale primitive inference attention network and integrated to obtain the energy map.

[0019] Furthermore, the step of performing convolution processing on the preprocessed trimodal medical image based on the first multi-scale compression and excitation inference attention module and sending the convolutional feature map to convolution branches of different scales specifically includes:

[0020] The preprocessed trimodal medical images are convolved to obtain a global attention map.

[0021] Information is collected and processed based on the global attention graph to obtain the global descriptor operator;

[0022] Primitive relation reasoning is performed based on global description operators to obtain an enhanced primitive relation reasoning attention map;

[0023] The enhanced primitive relation reasoning attention map is used to assign features to the preprocessed trimodal medical image to obtain the output results.

[0024] Furthermore, the step of collecting and processing information based on the global attention map to obtain the global descriptive operator specifically includes:

[0025] Dimension reduction preprocessing is performed on the preprocessed trimodal medical image and global attention map respectively to obtain the corresponding low-dimensional preprocessed trimodal medical image and low-dimensional global attention map;

[0026] Bilinear pooling is performed on the low-dimensional preprocessed trimodal medical image and the low-dimensional global attention map to obtain the global descriptor operator.

[0027] Furthermore, the step of performing primitive relation reasoning based on global description operators to obtain an enhanced primitive relation reasoning attention map specifically includes:

[0028] The global description operator is embedded into two embedding spaces with different weight parameters, and the affinity matrix is ​​obtained by calculating pairwise affinity.

[0029] A graph convolutional network is used to infer the relationships between descriptive operators in the affinity matrix, generating a primitive relation inference attention graph.

[0030] The global descriptor is enhanced by generating a primitive relation inference attention map, resulting in an enhanced primitive relation inference attention map.

[0031] Furthermore, the step of assigning features from the enhanced primitive relation reasoning attention map to the preprocessed trimodal medical image and obtaining the output result specifically includes:

[0032] The preprocessed trimodal medical images are transformed to generate learnable attention vector maps;

[0033] The global information in the enhanced primitive relation reasoning attention map is used to assign features to a learnable attention vector map to obtain the output result.

[0034] Furthermore, the step of combining the preprocessed trimodal medical images with the corresponding energy maps, and then performing fusion processing on the combination results using an energy ratio fusion strategy to output a preliminary fused image, specifically includes:

[0035] The preprocessed trimodal medical images are processed using an energy ratio fusion strategy to generate corresponding fusion coefficients;

[0036] Based on the fusion coefficient, the preprocessed trimodal medical images are combined with the corresponding energy maps to output a preliminary fused image.

[0037] Furthermore, the step of constructing an adversarial game relationship model based on the discriminator and generator to perform preliminary discrimination processing on the fused image, generating a fused image containing more information from the third type of source image, and obtaining the final fused image, specifically includes:

[0038] An adversarial game relationship model is constructed based on a discriminator and a generator. The adversarial game relationship model includes a three-layer network structure and a sigmoid function. The first layer of the network structure includes a 3×3 convolutional layer and a ReLU activation function. The second layer of the network structure includes a 3×3 convolutional layer, a BN layer and a ReLU activation function. The third layer of the network structure includes an adaptive max pooling layer and a 1×1 convolutional layer.

[0039] Based on the first-layer network structure of the adversarial game relationship model, feature extraction is performed on the preliminary fused image to obtain multiple feature maps;

[0040] The second-layer network structure based on the adversarial game relationship model performs secondary extraction of feature maps and adds BN and ReLU to accelerate training speed.

[0041] The third-layer network structure based on the adversarial game relationship model performs max pooling on the feature map and finally uses a 1×1 convolutional layer to calculate the scalar.

[0042] Based on the adversarial game relationship model, the sigmoid function maps the scalar output of the third-layer network structure to the [0,1] interval;

[0043] The second technical solution adopted in this invention is: a three-modal medical image fusion system based on primitive reasoning networks, comprising:

[0044] The preprocessing module is used to acquire trimodal medical images and perform image preprocessing to obtain preprocessed trimodal medical images, wherein the trimodal medical images include a first type of source image, a second type of source image, and a third type of source image;

[0045] The computation module performs energy calculation on the preprocessed trimodal medical images based on a multi-scale primitive inference attention network to obtain the corresponding energy map;

[0046] The fusion module is used to combine the preprocessed trimodal medical images with the corresponding energy maps, and to perform fusion processing on the combination results through an energy ratio fusion strategy to output a preliminary fused image.

[0047] The initial fused image is a grayscale image. The present invention also needs to combine the grayscale fused image with the data from the CR and CB channels of the second type of source image to generate a color fused image.

[0048] The discrimination module constructs an adversarial game relationship model based on the discriminator and generator to perform preliminary discrimination processing on the fused image, generate a fused image containing more information from the third type of source image, and obtain the final fused image.

[0049] The beneficial effects of the method and system of this invention are as follows: This invention proposes an end-to-end trimodal image fusion framework using primitive relation reasoning. It generates an energy map for each source image through a multi-scale excitation and compressed inference attention network. Finally, under the guidance of a capability ratio strategy, it achieves efficient trimodal medical image fusion. Furthermore, to obtain and establish global contextual relationships, it introduces squeezed and excitation inference attention blocks and simultaneously uses primitive relation reasoning to enhance global features. In summary, this invention fuses different and complementary information from original multimodal medical images into one image, which can help doctors better analyze difficult and complex diseases and reduce misdiagnosis and surgical errors. Attached Figure Description

[0050] Figure 1 This is a flowchart of the steps of the trimodal medical image fusion method based on primitive reasoning network of the present invention;

[0051] Figure 2 This is a structural block diagram of the trimodal medical image fusion system based on primitive reasoning networks of the present invention;

[0052] Figure 3 This is a flowchart illustrating the image fusion algorithm proposed in this invention;

[0053] Figure 4 This is a schematic diagram of the multi-scale squeezing and excitation inference attention network structure of the present invention;

[0054] Figure 5 This is a schematic diagram of the squeeze and excitation reasoning attention block structure of the present invention;

[0055] Figure 6 This is a schematic diagram illustrating the process of primitive relation reasoning in this invention;

[0056] Figure 7 This is a schematic diagram of the structure of the adversarial game relationship model constructed based on the discriminator and generator of the present invention;

[0057] Figure 8 This is a schematic diagram showing the results of comparing the method of the present invention with two recent and classic image fusion methods. Detailed Implementation

[0058] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0059] Reference Figure 1 and Figure 3 This invention provides a three-modal medical image fusion method based on primitive reasoning networks, which includes the following steps:

[0060] S1. Acquire trimodal medical images and perform image preprocessing to obtain preprocessed trimodal medical images, wherein the trimodal medical images include a first type of source image, a second type of source image, and a third type of source image;

[0061] Specifically, medical images of different modalities are categorized, among which... The class diagram is MR-T2. Classification images include PET and SPECT. The types of images include MR-Gad, MR-T1, and CT. First, let's... Convert the class diagram to a YCRCB image, and then extract the Y channel. Class diagram, Class diagram and The images are input into a multi-scale primitive inference network with shared parameters.

[0062] S2. Based on a multi-scale primitive inference attention network, energy calculation is performed on the preprocessed trimodal medical images to obtain the corresponding energy maps;

[0063] Specifically, refer to Figure 4 and Figure 5 To ensure consistency in the energy map computation method, the proposed framework for different source images adopts a shared weight approach to form the generator. The multi-scale squeezing and activation network is divided into two parts: one part consists of a multi-scale module with residual connections and a squeezing and activation inference block, and the other part consists of a multi-scale module, a squeezing and activation inference attention module, and a convolutional layer.

[0064] The preprocessed trimodal medical images are input into a multi-scale primitive relation reasoning attention network, which includes a first squeezing and excitation reasoning attention block, a multi-scale module with residual connections, a second multi-scale block, a squeezing and excitation reasoning attention module, and a convolutional layer.

[0065] The first multi-scale compression and activation inference attention module performs convolution processing on the preprocessed trimodal medical image and sends the resulting feature maps to convolution branches at different scales. The multi-scale module with residual connections extracts multi-scale features from the source image, obtaining multiple multi-scale feature maps. The compression and activation inference attention block primarily extracts global semantic information from the multi-scale feature maps, generating multiple feature maps with both global and local features. Finally, element-wise addition is performed with the input feature map from the multi-scale module with residual connections.

[0066] The second multi-scale module performs convolution processing on the output of the previous part and sends the resulting feature maps to convolution branches of different scales. The multi-scale module extracts multi-scale features from the feature maps, resulting in multiple multi-scale feature maps. The squeezing and stimulated inference attention block primarily extracts global semantic information from the multi-scale feature maps, generating multiple feature maps with both global and local features. In fact, the second multi-scale module and the squeezing and stimulated inference attention block function similarly to the first part; their multiple uses are mainly to enhance the model's representational power, thereby improving the model's performance and effectiveness in deep learning tasks. Finally, all the obtained feature maps are input into the final convolutional layer and integrated to obtain the energy map.

[0067] In this invention, feature maps obtained from convolutional layers with different receptive fields are merged by cascading along the channel dimension. The multi-scale module first performs a 3×3 convolution on the input image and feeds the resulting feature map into two convolutional branches at different scales. For each branch, the first convolutional layer is a 1×1 convolutional kernel to reduce the number of channels. Subsequently, each layer is followed by a different number of 3×3 convolutional layers. Finally, feature maps at different scales are merged along the channels. In addition, a BN layer is added after each convolutional layer to accelerate network training.

[0068] An adversarial game relationship model is constructed based on a discriminator and a generator. The adversarial game relationship model includes a three-layer network structure and a sigmoid function. The first layer of the network structure includes a 3×3 convolutional layer and a ReLU activation function. The second layer of the network structure includes a 3×3 convolutional layer, a BN layer and a ReLU activation function. The third layer of the network structure includes an adaptive max pooling layer and a 1×1 convolutional layer.

[0069] The first layer of the network, based on the adversarial game relationship model, extracts features from the initial fused image to obtain multiple feature maps. The second layer of the network, also based on the adversarial game relationship model, performs secondary extraction on the feature maps and adds BN and ReLU to accelerate training. The third layer of the network, based on the adversarial game relationship model, performs max pooling on the feature maps and finally uses a 1×1 convolutional layer to calculate the scalar. The sigmoid function, based on the adversarial game relationship model, maps the scalar output of the third layer of the network to the [0,1] interval.

[0070] S21, Multi-scale squeezing and incentive-based reasoning attention module;

[0071] Specifically, the squeezed and incentivized inference attention block workflow mainly consists of three steps: information gathering, primitive inference, and feature allocation.

[0072] The purpose of the first step, information gathering, is to obtain the global descriptor operator. Before bilinear pooling, the input graph A∈ To obtain the global attention map, we need to pass through a convolutional layer. Then, dimensionality reduction preprocessing is performed on the input graph A and the global attention graph M respectively to make... and Therefore, a single descriptor is obtained by using bilinear pooling. The formula is as follows:

[0073] ;

[0074] In its implementation, this invention uses the sofxmax function to guarantee... Then, the present invention further describes the global information collection process using the following formula:

[0075] ;

[0076] After obtaining the global descriptor P, this paper describes each descriptor... The relationship between P and P is enhanced using PRR to obtain the enhanced global descriptor operator. Specifically, the global descriptor P is embedded into two objects with weight parameters. and Embedded space and Then, the affinity matrix is ​​obtained by calculating pairwise affinity to establish relationships. The formula is as follows:

[0077] ;

[0078] Using the above formula, this invention obtains the relationships between all two descriptive operators and generates a fully connected relationship graph;

[0079] Reference Figure 6 By using a graph convolutional network to perform relational reasoning on the global descriptor, a primitive relational reasoning attention map can be obtained. This map is then used to enhance the global descriptor. The specific formula is as follows:

[0080] ;

[0081] here It is the sigmoid activation function. It is a C×C weight matrix in the GCN layer. It is a C×C weight matrix in the residual connection.

[0082] The final step is feature allocation, which aims to assign the enhanced global descriptor to the original input feature map to obtain the output result Z. To enable each location in the original input feature map to adaptively select complementary descriptors, this invention transforms the original input feature map A into a learnable attention vector map through a convolutional layer. The feature allocation formula is as follows:

[0083] ;

[0084] here ∈ Z=[ ,∙∙∙, ]∈ The i-th column vector. In the implementation, this invention uses the softmax function to guarantee... Then, the present invention further describes the adaptive global information allocation process using the following formula:

[0085] ;

[0086] The final output is Z∈ It is also necessary to perform an expansion dimension operation to obtain Z∈ .

[0087] S3. Combine the preprocessed trimodal medical images with the corresponding energy maps, and perform fusion processing on the combination results using an energy ratio fusion strategy to output a preliminary fused image;

[0088] Specifically, refer to Figure 3 The energy diagram obtained from step S2 , and and source image , , A fused image F is generated using an energy ratio strategy. Specifically, this paper generates an energy map by inputting three source images into the same multi-scale compression and incentive inference attention network. , and Then, an energy ratio strategy is used to generate three corresponding fusion coefficients for these three energy maps. , and Finally, these three source images are combined to generate a fused image. The formula for the fusion coefficient is expressed as follows:

[0089] ;

[0090] ;

[0091] Here x∈A,B,C

[0092] S4. Based on the discriminator and generator, an adversarial game relationship model is constructed to perform preliminary discrimination processing on the fused image, generate a fused image containing more information from the third type of source image, and obtain the final fused image.

[0093] Specifically, this is an adversarial game between a generator G and a discriminator D. More specifically, the discriminator D aims to estimate the input image as... Source image or fused image The probability of the generator is determined by the number of real numbers, and the generator's goal is to generate as many real numbers as possible. Fusion image of source image information This invention aims to deceive the discriminator and ensure that the discriminator can accurately identify the target during the adversarial process. Source images and fused images The generator can produce a sufficiently realistic fused image that can fool the discriminator. In summary, this can be summarized by the following formula:

[0094] ;

[0095] Since this paper studies trimodal medical image fusion, its fusion process is more complex than that of two-modal image fusion. This paper incorporates a discriminator D into the proposed method, establishing an adversarial game relationship between the discriminator and the generator, aiming to make the fused image contain more... Useful information about images, such as Figure 7 As shown, the first two layers both contain convolutional layers with 3×3 kernels and the ReLU activation function. The second layer incorporates Batch Normalization to accelerate training. The third layer uses an adaptive max pooling layer and a convolutional layer with 1×1 kernels. Finally, a scalar is output through the sigmoid function.

[0096] The obtained fusion image under the Y channel With source image , and the source image under the Y channel Calculate the generator's loss function. Generator Loss It mainly includes two items: structural similarity loss. and adversarial losses .

[0097] ;

[0098] for The formula is expressed as follows:

[0099] ;

[0100] ;

[0101] x∈A,B,C, and Representing the source medical images The intensity of the fused multimodal medical image F, and Indicates two constants, and Representing the source medical images and fused multimodal medical images variance Represents source medical images and fused multimodal medical images The covariance.

[0102] For a discriminator The formula is expressed as follows:

[0103] ;

[0104] This indicates that the discriminator considers the fused image to be... The probability of.

[0105] Discriminator loss The formula is defined as follows:

[0106] ;

[0107] D( This indicates that the discriminator considers the source image to be... yes The probability of.

[0108] Reference Figure 2A trimodal medical image fusion system based on primitive reasoning networks includes:

[0109] The preprocessing module is used to acquire trimodal medical images and perform image preprocessing to obtain preprocessed trimodal medical images, wherein the trimodal medical images include a first type of source image, a second type of source image, and a third type of source image;

[0110] The computation module performs energy calculation on the preprocessed trimodal medical images based on a multi-scale primitive inference attention network to obtain the corresponding energy map;

[0111] The fusion module is used to combine the preprocessed trimodal medical images with the corresponding energy maps, and to perform fusion processing on the combination results through an energy ratio fusion strategy to output a preliminary fused image.

[0112] The discrimination module constructs an adversarial game relationship model based on the discriminator and generator to perform preliminary discrimination processing on the fused image, generate a fused image containing more information from the third type of source image, and obtain the final fused image.

[0113] Experimental results based on the method proposed in this invention are as follows:

[0114] In both sets of fusion results, EMFusion exhibits low contrast. In the first set of fusion results, the reinforcement learning fusion method displays white noise, while the proposed method eliminates white noise while maintaining high color fidelity. Regarding the second set of fusion results, magnifying the region reveals that, compared to the other two methods, the proposed method retains both useful MR-T2 information and edge detail information of MR-T1. In conclusion, compared to the other two methods, EMFusion demonstrates a strong ability to eliminate redundant information and preserve source image information in this type of trimodal image fusion.

[0115] Table 1 shows the average values ​​of each fusion index for 30 sets of trimodal source image pairs.

[0116] Methods EN SSIM QCB SD PSNR QG This invention application 0.9865 0.5334 0.3434 0.4891 55.2184 0.3817 Multi-scale fusion 0.9756 0.5066 0.3226 0.4877 55.0141 0.3697 Enhanced learning integration 0.9801 0.5164 0.3128 0.4847 54.8843 0.3659

[0117] As shown in Table 1, the proposed method achieved good scores on all six metrics, specifically SD, , , , and All six indicators ranked first in average score, and combined with the proposed method... Figure 8The subjective evaluations show that the proposed method outperforms the two comparative methods in terms of robustness, human visual effect, edge information, structural information, and information content.

[0118] In summary, this invention categorizes medical images of different modalities, wherein... The class diagram is MR-T2. Classification images include PET and SPECT. The types of images include MR-Gad, MR-T1, and CT. First, let's... The class diagram is converted to a YCRCB image. Then the Y channel is... Class diagram, Class diagram and The energy maps are obtained by inputting the images into a multi-scale primitive inference network with shared parameters. , and Combined with source image , , and energy diagram , and The fusion map is obtained through the energy ratio fusion strategy. .

[0119] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0120] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A three-modal medical image fusion method based on primitive reasoning networks, characterized in that, Includes the following steps: A trimodal medical image is acquired and preprocessed to obtain a preprocessed trimodal medical image, wherein the trimodal medical image includes a first type of source image, a second type of source image, and a third type of source image; The preprocessed trimodal medical images are input into a multi-scale primitive relation inference attention network, which includes a first squeezing and excitation attention block, a multi-scale module with residual connections, a second multi-scale block, a squeezing and excitation inference attention module, and a convolutional layer. The first squeezing and excitation attention block performs convolution processing on the preprocessed trimodal medical image and sends the convolutional feature map to convolution branches of different scales. The multi-scale module with residual connections is used to extract multi-scale features of the source image to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature maps to generate multiple feature maps with global and local features. The feature maps are then added element-wise with the input feature map of the multi-scale module with residual connections to obtain the first output result. The first output result is convolved based on the second multi-scale block, and the feature map obtained by convolution is sent to convolution branches of different scales. The multi-scale module is used to extract multi-scale features of the feature map to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature map to generate multiple feature maps with global and local features. All the obtained feature maps are input into the final convolutional layer of the multi-scale primitive inference attention network and integrated to obtain the energy map; The preprocessed trimodal medical images are combined with the corresponding energy maps, and the combination results are fused using an energy ratio fusion strategy to output a preliminary fused image. An adversarial game relationship model based on a discriminator and a generator is constructed to perform preliminary discrimination processing on the fused image, generating a fused image containing more information from the third type of source image, and obtaining the final fused image.

2. The trimodal medical image fusion method based on primitive reasoning networks according to claim 1, characterized in that, The step of acquiring trimodal medical images and performing image preprocessing to obtain preprocessed trimodal medical images specifically includes: Acquire trimodal medical images, which include a first type of source image, a second type of source image, and a third type of source image. The first type of source image includes MR-T2 images, the second type of source image includes PET images and SPECT images, and the third type of source image includes MR-Gad images, MR-T1 images, and CT images. Convert the second type of source image into a YCRCB image to obtain the second type of source image in the Y channel; By combining the first type of source image, the second type of source image under the Y channel, and the third type of source image, a preprocessed trimodal medical image is constructed.

3. The trimodal medical image fusion method based on primitive reasoning networks according to claim 2, characterized in that, The step of performing convolution processing on the preprocessed trimodal medical image based on the first multi-scale compression and excitation inference attention module and sending the convolutional feature map to convolution branches of different scales specifically includes: The preprocessed trimodal medical images are convolved to obtain a global attention map. Information is collected and processed based on the global attention graph to obtain the global descriptor operator; Primitive relation reasoning is performed based on global description operators to obtain an enhanced primitive relation reasoning attention map; The enhanced primitive relation reasoning attention map is used to assign features to the preprocessed trimodal medical image to obtain the output result, which is a multi-feature map with both global and local features.

4. The trimodal medical image fusion method based on primitive reasoning networks according to claim 3, characterized in that, The step of collecting and processing information based on the global attention map to obtain the global descriptor operator specifically includes: Dimension reduction preprocessing is performed on the preprocessed trimodal medical image and global attention map respectively to obtain the corresponding low-dimensional preprocessed trimodal medical image and low-dimensional global attention map; Bilinear pooling is performed on the low-dimensional preprocessed trimodal medical image and the low-dimensional global attention map to obtain the global descriptor operator.

5. The trimodal medical image fusion method based on primitive reasoning networks according to claim 4, characterized in that, The step of performing primitive relation reasoning based on global description operators and obtaining an enhanced primitive relation reasoning attention map specifically includes: The global description operator is embedded into two embedding spaces with different weight parameters, and the affinity matrix is ​​obtained by calculating pairwise affinity. A graph convolutional network is used to infer the relationships between descriptive operators in the affinity matrix, generating a primitive relation inference attention graph. The global descriptor is enhanced by generating a primitive relation inference attention map, resulting in an enhanced primitive relation inference attention map.

6. The trimodal medical image fusion method based on primitive reasoning networks according to claim 5, characterized in that, The step of assigning features from the enhanced primitive relation reasoning attention map to the preprocessed trimodal medical image and obtaining the output result specifically includes: The preprocessed trimodal medical images are transformed to generate learnable attention vector maps; The global information in the enhanced primitive relation reasoning attention map is used to assign features to a learnable attention vector map to obtain the output result.

7. The trimodal medical image fusion method based on primitive reasoning networks according to claim 6, characterized in that, The step of combining the preprocessed trimodal medical images with the corresponding energy maps, and then performing fusion processing on the combination results using an energy ratio fusion strategy to output a preliminary fused image, specifically includes: The preprocessed trimodal medical images are processed using an energy ratio fusion strategy to generate corresponding fusion coefficients; Based on the fusion coefficient, the preprocessed trimodal medical images are combined with the corresponding energy maps to output a preliminary fused image.

8. The trimodal medical image fusion method based on primitive reasoning networks according to claim 7, characterized in that, The step of constructing an adversarial game relationship model based on the discriminator and generator to perform preliminary discrimination processing on the fused image, generating a fused image containing more information from the third type of source image, and obtaining the final fused image, specifically includes: An adversarial game relationship model is constructed based on a discriminator and a generator. The adversarial game relationship model includes a three-layer network structure and a sigmoid function. The first layer of the network structure includes a 3×3 convolutional layer and a ReLU activation function. The second layer of the network structure includes a 3×3 convolutional layer, a BN layer and a ReLU activation function. The third layer of the network structure includes an adaptive max pooling layer and a 1×1 convolutional layer. Based on the first-layer network structure of the adversarial game relationship model, feature extraction is performed on the preliminary fused image to obtain multiple feature maps; The second-layer network structure based on the adversarial game relationship model performs secondary extraction of feature maps and adds BN and ReLU to accelerate training speed. The third-layer network structure based on the adversarial game relationship model performs max pooling on the feature map and finally uses a 1×1 convolutional layer to calculate the scalar. The sigmoid function based on the adversarial game relationship model maps the scalar output of the third-layer network structure to the [0,1] interval.

9. A trimodal medical image fusion system based on primitive reasoning networks, characterized in that, Includes the following modules: The preprocessing module is used to acquire trimodal medical images and perform image preprocessing to obtain preprocessed trimodal medical images, wherein the trimodal medical images include a first type of source image, a second type of source image, and a third type of source image; The computation module inputs the preprocessed trimodal medical images into a multi-scale primitive relation inference attention network, which includes a first squeezing and excitation attention block, a multi-scale module with residual connections, a second multi-scale block, a squeezing and excitation inference attention module, and a convolutional layer. The first squeezing and excitation attention block performs convolution processing on the preprocessed trimodal medical image and sends the convolutional feature map to convolution branches of different scales. The multi-scale module with residual connections is used to extract multi-scale features of the source image to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature maps to generate multiple feature maps with global and local features. The feature maps are then added element-wise with the input feature map of the multi-scale module with residual connections to obtain the first output result. The first output result is convolved based on the second multi-scale block, and the feature map obtained by convolution is sent to convolution branches of different scales. The multi-scale module is used to extract multi-scale features of the feature map to obtain multiple multi-scale feature maps. The squeezing and excitation inference attention block is used to extract global semantic information from the multi-scale feature map to generate multiple feature maps with global and local features. All the obtained feature maps are input into the final convolutional layer of the multi-scale primitive inference attention network and integrated to obtain the energy map; The fusion module is used to combine the preprocessed trimodal medical images with the corresponding energy maps, and to perform fusion processing on the combination results through an energy ratio fusion strategy to output a preliminary fused image. The discrimination module constructs an adversarial game relationship model based on the discriminator and generator to perform preliminary discrimination processing on the fused image, generate a fused image containing more information from the third type of source image, and obtain the final fused image.

Images