Medical image fusion method and system based on correlation constraint and attention guidance

Abstract

The invention provides a medical image fusion method and system based on correlation constraint and attention guidance, which can be applied to the technical field of medical image processing. The method comprises the following steps: performing multi-level feature extraction on a channel splicing image, a preprocessed MRI image and a preprocessed CT image by using a feature extraction module with correlation constraints to obtain multi-level features of the MRI image, multi-level features of the CT image and multi-level features of the channel splicing image; performing double-layer cross attention feature fusion on the MRI image multi-level features, the CT image multi-level features and the channel splicing image multi-level features on each common level by using a feature fusion module with an attention guiding mechanism to obtain multi-level fusion features with different channel numbers; and splicing the multi-level fusion features with different channel numbers by using a feature reconstruction module of the medical image fusion network, and carrying out multi-modal medical image reconstruction on the channel splicing fusion features to obtain a medical image fusion result.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly to a medical image fusion method and system based on correlation constraint and attention guidance. Background Art

[0002] Due to the inherent limitations of imaging technologies, a single medical imaging modality can only capture specific tissue characteristics. For example, Computed Tomography (CT) can clearly image bones and implanted objects in the body, but it is difficult to capture detailed information of other organ tissues; Magnetic Resonance (MR) can clearly image organs such as soft tissues, but it is difficult to reflect the characteristics of the bone region. In the diagnosis and evaluation of tumors, the images obtained by these different devices contain a large amount of complementary information. The purpose of multimodal medical image fusion is to integrate image information of different modalities into a single image, thereby supporting tasks such as lesion detection and internal tissue and organ analysis. Therefore, multimodal medical image fusion is becoming increasingly important in many medical applications such as disease detection, surgical navigation, and treatment design, because it can provide complementary insights into tissue and pathological changes through different imaging modalities, thereby supporting tasks such as lesion detection and internal tissue and organ analysis. The fusion of MRI and CT images can generate images with clear MR textures and CT bone implant information, greatly improving clinical diagnostic decision-making and diagnostic accuracy, and has become an important tool for assisting doctors in diagnosis and treatment.

[0003] Currently, the existing deep learning-based image fusion methods have become the mainstream methods in the field of medical image fusion. Generally, deep neural networks are used to extract multimodal features, and cross-modal feature interaction is used to achieve feature fusion. However, there are still two problems as follows: 1): There is insufficient constraint between multimodal feature extraction branches, which may lead to imbalance in extracting shared information between modalities and specific information within modalities. 2): Existing methods often only consider the fusion between specific features of different modalities, ignoring the interactive fusion of common features and unique features in multimodal features, resulting in reduced utilization rate of multimodal information, which limits the fusion effect. Summary of the Invention

[0004] In view of the above problems, the present invention provides a medical image fusion method and system based on correlation constraint and attention guidance, which is used to solve at least one of the above problems.

[0005] According to a first aspect of the present invention, there is provided a medical image fusion method based on correlation constraint and attention guidance, including:

[0006] Perform channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain a channel-stitched image;

[0007] Use the feature extraction module with correlation constraints in the medical image fusion network to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image, obtaining multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image;

[0008] Use the feature fusion module with an attention guidance mechanism in the medical image fusion network to perform feature fusion of double-layer cross-attention on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-stitched image at each common level, obtaining multi-level fusion features with different numbers of channels;

[0009] Use the feature reconstruction module of the medical image fusion network to perform channel stitching on the multi-level fusion features with different numbers of channels to obtain a channel-stitched fusion feature, and perform multi-modal medical image reconstruction on the channel-stitched fusion feature to obtain a medical image fusion result.

[0010] According to an embodiment of the present invention, the above-mentioned performing channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain a channel-stitched image includes:

[0011] Perform pixel value normalization processing on the MRI image and the CT image respectively to obtain the preprocessed MRI image and the preprocessed CT image;

[0012] Perform channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain a channel-stitched image.

[0013] According to an embodiment of the present invention, the above-mentioned using the feature extraction module with correlation constraints in the medical image fusion network to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image, obtaining multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image includes:

[0014] Use the MRI feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed MRI image to obtain multi-level features of the MRI image;

[0015] Use the CT feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed CT image to obtain multi-level features of the CT image;

[0016] The preprocessed channel concatenated image is subjected to convolution processing and multiple spatial-channel convolution processes by the common feature extraction module of the medical image fusion network to obtain multi-level features of the channel concatenated image. Among them, there is a correlation constraint among the MRI feature extraction module, the CT feature extraction module, and the common feature extraction module.

[0017] According to an embodiment of the present invention, the above-mentioned use of the MRI feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processes on the preprocessed MRI image to obtain multi-level features of the MRI image includes:

[0018] The preprocessed MRI image is converted to the feature domain by the initial convolution layer of the MRI feature extraction module to obtain initial convolution features;

[0019] The initial convolution features are subjected to multiple spatial convolutions and channel convolutions by multiple spatial-channel convolution layers of the MRI feature extraction module to obtain multi-level spatial-channel convolution features with different numbers of channels. Among them, the multi-level features of the MRI image include the initial convolution features and the multi-level spatial-channel convolution features with different numbers of channels.

[0020] According to an embodiment of the present invention, the above-mentioned use of multiple spatial-channel convolution layers of the MRI feature extraction module to perform multiple spatial convolutions and channel convolutions on the initial convolution features to obtain multi-level features of the MRI image includes:

[0021] The initial convolution features are subjected to convolution processing, group normalization processing, and multiple activation processes in the spatial dimension by the first spatial-channel convolution layer to obtain first spatial convolution features;

[0022] The first spatial convolution features are subjected to group convolution processing and pointwise convolution processing in the channel dimension in parallel, and the obtained first group convolution features and first pointwise convolution features are concatenated to obtain first spatial-channel convolution features;

[0023] The first spatial-channel convolution features are subjected to convolution processing, group normalization processing, and multiple activation processes in the spatial dimension by the second spatial-channel convolution layer to obtain second spatial convolution features;

[0024] The second spatial convolution features are subjected to group convolution processing and pointwise convolution processing in the channel dimension in parallel, and the obtained second group convolution features and second pointwise convolution features are concatenated to obtain second spatial-channel convolution features;

[0025] The second spatial-channel convolution features are subjected to convolution processing, group normalization processing, and multiple activation processes in the spatial dimension by the third spatial-channel convolution layer to obtain third spatial convolution features;

[0026] Perform grouped convolution processing and pointwise convolution processing on the third spatial convolution features in parallel in the channel dimension, and splice the obtained third grouped convolution features and third pointwise convolution features to obtain third spatial channel convolution features.

[0027] According to an embodiment of the present invention, the correlation constraint between the above-mentioned MRI feature extraction module, CT feature extraction module, and common feature extraction module is obtained through the following operations:

[0028] Iteratively train the medical image fusion network, and during the iterative training process of the medical image fusion network, reduce the redundancy of the MRI feature extraction module and the CT feature extraction module and improve the correlation between the common feature extraction module, the MRI feature extraction module, and the CT feature extraction module through the correlation constraint loss function in the preset loss function.

[0029] According to an embodiment of the present invention, the above-mentioned iterative training of the medical image fusion network includes:

[0030] Use the medical image fusion network to parallelly perform multi-level feature extraction on the preprocessed multi-modal medical image samples to obtain the multi-level features of the multi-modal medical image samples, and use the correlation constraint loss function to perform correlation constraints on the feature extraction process;

[0031] Use the medical image fusion network to perform feature fusion based on the double-layer cross-attention guidance mechanism on the multi-level features of the multi-modal medical image samples at each common level to obtain the multi-level fusion features of the multi-modal medical image samples;

[0032] Use the medical image fusion network to splice the multi-level fusion features of the multi-modal medical image samples, and perform image reconstruction on the spliced features of the multi-modal medical image samples to obtain the medical fusion result of the multi-modal medical image samples;

[0033] Use the preset loss function to process the multi-modal medical image samples and the medical image fusion result of the multi-modal medical image samples to obtain a loss value, and update the parameters of the medical image fusion network based on the loss value;

[0034] Iteratively perform operations of feature extraction, feature fusion, image reconstruction, loss value calculation, and parameter update until the preset training conditions are met to obtain a trained medical image fusion network.

[0035] According to an embodiment of the present invention, the above-mentioned use of the feature fusion module with an attention guidance mechanism in the medical image fusion network to perform double-layer cross-attention feature fusion on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-spliced image at each common level to obtain multi-level fusion features with different numbers of channels includes:

[0036] Use the multi-level features of the channel-spliced image as common features to generate multi-level query vectors in the attention guidance mechanism;

[0037] Use the multi-level features of the MRI image as specific features to generate MRI multi-level key vectors and MRI multi-level value vectors in the attention guidance mechanism;

[0038] Use the multi-level features of the CT image as specific features to generate CT multi-level key vectors and CT multi-level value vectors in the attention guidance mechanism;

[0039] At each common level, use the common feature fusion module of the medical image fusion network to perform cross-attention feature fusion guided by common features on the multi-level query vectors, MRI multi-level key vectors, MRI multi-level value vectors, CT multi-level key vectors, and CT multi-level value vectors to obtain multi-level common fusion features with different numbers of channels.

[0040] According to an embodiment of the present invention, the above-mentioned feature fusion module with an attention guidance mechanism in the medical image fusion network performs double-layer cross-attention feature fusion on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-spliced image at each common level to obtain multi-level fusion features with different numbers of channels, further including:

[0041] Use the multi-level features of the channel-spliced image as common features to generate multi-level key vectors and multi-level value vectors in the attention guidance mechanism;

[0042] Use the multi-level features of the MRI image as specific features to generate MRI multi-level query vectors in the attention guidance mechanism;

[0043] Use the multi-level features of the CT image as specific features to generate CT multi-level query vectors in the attention guidance mechanism;

[0044] At each common level, use the specific feature fusion module of the medical image fusion network to perform cross-attention feature fusion guided by specific features on the multi-level key vectors, multi-level value vectors, MRI multi-level query vectors, and CT multi-level query vectors to obtain multi-level specific fusion features with different numbers of channels;

[0045] Perform element-wise operations on the multi-level common fusion features with different numbers of channels and the multi-level specific fusion features with different numbers of channels on the same number of channels to obtain multi-level fusion features with different numbers of channels.

[0046] The second aspect of the present invention provides a medical image fusion system based on correlation constraint and attention guidance, including:

[0047] Channel stitching image acquisition module, which is used to stitch the preprocessed MRI image and the preprocessed CT image in the channel dimension to obtain a channel-stitched image;

[0048] Multi-level feature extraction module, which is used to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image by using the feature extraction module with correlation constraints in the medical image fusion network to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image;

[0049] Multi-channel feature fusion module, which is used to perform feature fusion of double-layer cross-attention on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-stitched image at each common level by using the feature fusion module with an attention guidance mechanism in the medical image fusion network to obtain multi-level fusion features with different numbers of channels;

[0050] Medical image fusion and reconstruction module, which is used to stitch the multi-level fusion features with different numbers of channels in the channel dimension by using the feature reconstruction module of the medical image fusion network to obtain a channel-stitched fusion feature, and perform multi-modal medical image reconstruction on the channel-stitched fusion feature to obtain a medical image fusion result.

[0051] The medical image fusion method and system based on correlation constraint and attention guidance provided by the present invention fully integrate the complementary information of the two modalities by stitching MRI and CT images in the channel dimension and combining multi-level feature extraction; at the same time, the correlation constraint is introduced into the feature extraction module of the medical image fusion network, which can effectively enhance the inter-modal correlation features, suppress redundant information, realize information sharing between different types of medical images and their stitched images, and reduce fusion noise; in addition, the feature fusion module of the medical image fusion network dynamically allocates weights through the double-layer cross-attention mechanism, that is, the unique features and common features of multi-modal medical images interact and guide the fusion, so that the fusion process focuses on the key areas of different modalities, realizes more refined fusion, and reduces the loss of important information. The medical image fusion method and system based on correlation constraint and attention guidance provided by the present invention can provide high-quality multi-modal medical fusion images for relevant scientific researchers or medical personnel, and provide more accurate medical auxiliary information for relevant scientific researchers or medical personnel. Description of the Drawings

[0052] Through the following description of the embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0053] Figure 1 It is an application scenario diagram of the medical image fusion method based on correlation constraint and attention guidance according to the embodiment of the present invention;

[0054] Figure 2 is a flowchart of a medical image fusion method based on correlation constraint and attention guidance according to an embodiment of the present invention;

[0055] Figure 3 is a schematic structural diagram of a medical image fusion network according to an embodiment of the present invention;

[0056] Figure 4 is a schematic structural diagram of a spatial and channel convolution unit according to an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of a multi-modal feature fusion process based on double-layer cross-attention guidance according to an embodiment of the present invention;

[0058] Figure 6 is a schematic diagram of a multi-modal image processing process according to an embodiment of the present invention;

[0059] Figure 7 is a comparative effect diagram of a multi-type multi-modal medical image fusion method according to an embodiment of the present invention;

[0060] Figure 8 is a schematic structural diagram of a medical image fusion system based on correlation constraint and attention guidance according to an embodiment of the present invention;

[0061] Figure 9 is a block diagram of an electronic device suitable for implementing a medical image fusion method based on correlation constraint and attention guidance according to an embodiment of the present invention. Detailed implementation manners

[0062] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, obviously, one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present invention.

[0063] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0064] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0065] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0066] Existing multi-modal medical image fusion technical solutions have the following technical problems: (1) There is insufficient constraint between multi-modal feature extraction branches, which may lead to an imbalance in the extraction of shared information between modalities and specific information within modalities; (2) Existing methods often only consider the fusion of specific features between different modalities, ignoring the interactive fusion of common features and unique features in multi-modal features, resulting in a reduction in the utilization rate of multi-modal information, which limits the fusion effect.

[0067] To solve the above technical problems, the present invention provides a medical image fusion method based on correlation constraint and attention guidance in the field of medical image processing technology. For the above technical problem (1), the method provided by the present invention introduces linear and non-linear correlation loss functions during the training process of the medical image fusion network, obtaining a feature extraction module with correlation constraint, which is used to achieve more sufficient constraint of the multi-modal feature extraction branch, help retain important information and reduce redundancy, and achieve a more balanced extraction of shared information between modalities and specific information within modalities; for the above technical problem (2), the method provided by the present invention utilizes a feature fusion module with guided attention concentration, which is used to achieve double-layer cross-attention fusion of multi-modal medical images, that is, to respectively focus on the interactive guided fusion of specific information and shared information, achieve more refined fusion, and reduce the loss of important information.

[0068] Figure 1 It is an application scenario diagram of the medical image fusion method based on correlation constraint and attention guidance according to an embodiment of the present invention.

[0069] As Figure 1 shown, the application scenario 100 according to this embodiment may include the field of medical image processing technology. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0070] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0071] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablets, laptop computers, desktop computers, and so on.

[0072] The server 105 can be a server that provides various services, such as a background management server that supports the websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background management server can analyze and process data such as received user requests, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0073] It should be noted that the medical image fusion method based on correlation constraint and attention guidance provided by the embodiments of the present invention can generally be executed by the server 105. Correspondingly, the medical image fusion system based on correlation constraint and attention guidance provided by the embodiments of the present invention can generally be set in the server 105. The medical image fusion method based on correlation constraint and attention guidance provided by the embodiments of the present invention can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the medical image fusion system based on correlation constraint and attention guidance provided by the embodiments of the present invention can also be set in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0074] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0075] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers. Figure 1 The following will be based on Figures 2 to 7A medical image fusion method based on correlation constraint and attention guidance for publicly disclosed embodiments is described in detail.

[0076] Figure 2 It is a flowchart of a medical image fusion method based on correlation constraint and attention guidance according to an embodiment of the present invention.

[0077] As Figure 2 shown, the medical image fusion method based on correlation constraint and attention guidance includes operations S210 to S240.

[0078] In operation S210, the preprocessed MRI image and the preprocessed CT image are stitched together on the channels to obtain a channel-stitched image.

[0079] For the preprocessing of the above-mentioned MRI image and CT image, the following operations are performed:

[0080] According to an embodiment of the present invention, the above-mentioned stitching of the preprocessed MRI image and the preprocessed CT image on the channels to obtain a channel-stitched image includes: performing pixel value normalization processing on the MRI image and the CT image respectively to obtain the preprocessed MRI image and the preprocessed CT image; stitching the preprocessed MRI image and the preprocessed CT image on the channels to obtain a channel-stitched image.

[0081] In operation S220, a feature extraction module with correlation constraint in the medical image fusion network is used to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image.

[0082] Among them, the above-mentioned multi-level features of the MRI image represent MRI image features with different numbers of channels.

[0083] The above-mentioned multi-level features of the CT image represent CT image features with different numbers of channels.

[0084] The above-mentioned multi-level features of the channel-stitched image represent channel-stitched image features with different numbers of channels.

[0085] The above-mentioned feature extraction module with correlation constraint includes three parallel branches, namely, a unique feature extraction module for the MRI image, a unique feature extraction module for the CT image, and a common feature extraction module.

[0086] During the training process of the medical image fusion network, a correlation constraint loss function is used to update the parameters of the above-mentioned multiple parallel feature extraction modules, so that the multiple feature extraction modules in the trained medical image fusion network have correlation constraints.

[0087] In operation S230, a feature fusion module with an attention guidance mechanism in the medical image fusion network performs double-layer cross-attention feature fusion on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel concatenation image at each common level, obtaining multi-level fusion features with different numbers of channels.

[0088] During the process of double-layer cross-attention feature fusion, generally, the MRT image features, CT image features, and channel concatenation features of the same-channel data are fused.

[0089] In operation S240, a feature reconstruction module of the medical image fusion network performs concatenation on the multi-level fusion features with different numbers of channels in the channel dimension, obtaining a channel concatenation fusion feature, and performs multi-modal medical image reconstruction on the channel concatenation fusion feature, obtaining a medical image fusion result.

[0090] The medical image fusion method and system based on correlation constraint and attention guidance provided by the present invention fully integrate the complementary information of the two modalities by concatenating the MRI and CT images in the channel dimension and combining multi-level feature extraction; meanwhile, the feature extraction module of the medical image fusion network introduces a correlation constraint, which can effectively enhance the inter-modal correlation features, suppress redundant information, realize information sharing between multi-type medical images and their concatenation images, and reduce fusion noise; in addition, the feature fusion module of the medical image fusion network dynamically allocates weights through a double-layer cross-attention mechanism, that is, the unique features and common features of multi-modal medical images interact and guide the fusion, making the fusion process focus on the key regions of different modalities, realizing a more refined fusion, and reducing the loss of important information. The above-mentioned medical image fusion method and system based on correlation constraint and attention guidance provided by the present invention can provide high-quality multi-modal medical fusion images for relevant scientific research personnel or medical personnel, and provide more accurate medical auxiliary information for relevant scientific research personnel or medical personnel.

[0091] According to an embodiment of the present invention, the above-mentioned multi-level feature extraction of the channel-spliced image, the preprocessed MRI image, and the preprocessed CT image by using the feature extraction module with correlation constraints in the medical image fusion network to obtain the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-spliced image includes: performing convolutional processing and multiple spatial-channel convolutional processes on the preprocessed MRI image by using the MRI feature extraction module of the medical image fusion network to obtain the multi-level features of the MRI image; performing convolutional processing and multiple spatial-channel convolutional processes on the preprocessed CT image by using the CT feature extraction module of the medical image fusion network to obtain the multi-level features of the CT image; performing convolutional processing and multiple spatial-channel convolutional processes on the preprocessed channel-spliced image by using the common feature extraction module of the medical image fusion network to obtain the multi-level features of the channel-spliced image, wherein there is a correlation constraint between the MRI feature extraction module, the CT feature extraction module, and the common feature extraction module.

[0092] In the above embodiment, three parallel feature extraction networks in the medical image fusion network are used to independently extract features from the MRI image, the CT image, and the channel-spliced image respectively, which not only preserves the specific information of each modality of medical images but also integrates the complementary information of the two modalities (i.e., the channel-spliced image). In addition, although each feature extraction module independently extracts features, during the training process of the medical image fusion network, the three feature extraction modules already have a correlation constraint. Therefore, the image features obtained by the above three feature extraction modules can effectively enhance the inter-modal correlation features (such as the consistency of organ contours, etc.), suppress redundant or conflicting information, ensure that the extracted features are more valuable for scientific research or medicine, and reduce the noise in the subsequent feature fusion process.

[0093] According to an embodiment of the present invention, the above-mentioned performing convolutional processing and multiple spatial-channel convolutional processes on the preprocessed MRI image by using the MRI feature extraction module of the medical image fusion network to obtain the multi-level features of the MRI image includes: converting the preprocessed MRI image to the feature domain by using the initial convolutional layer of the MRI feature extraction module to obtain the initial convolutional features; performing multiple spatial convolutions and channel convolutions on the initial convolutional features by using multiple spatial-channel convolutional layers of the MRI feature extraction module to obtain multi-level spatial-channel convolutional features with different numbers of channels, wherein the multi-level features of the MRI image include the initial convolutional features and the multi-level spatial-channel convolutional features with different numbers of channels.

[0094] In the above embodiment, local perception and spatial structure preservation of the MRI image can be achieved by performing convolutional processing and multiple spatial-channel convolutional processes, that is, through the sliding window mechanism of the convolutional kernel (such as Convolution) can effectively preserve the local spatial correlation of MRI images when extracting features. Compared with the problem of losing spatial information caused by the fully connected network expanding the image into a 1D vector, this solution avoids the destruction of anatomical structures (such as organ edges), which is more conducive to capturing the detailed textures in medical images (such as lesion areas); at the same time, the convolutional kernel shares parameters across the entire image, significantly reducing the number of model parameters. For example, the number of parameters of the convolutional kernel is only 9, while the parameters of a fully connected layer with the same receptive field may be as high as tens of thousands. This is particularly important for high-resolution medical images (such as MRI), which can prevent overfitting and improve training efficiency. In addition, through multiple spatial-channel convolutional processes, the receptive field is gradually expanded layer by layer, that is, the initial convolutional layer extracts low-level features, and the deep spatial-channel convolution captures high-level semantics (such as organ shape, global distribution of lesion areas). This multi-scale feature combination can retain both details and global information, enhancing the representation ability of complex anatomical structures.

[0095] The feature extraction process of CT images and channel-spliced images is similar to that of MRI images, which will not be elaborated here.

[0096] According to an embodiment of the present invention, the above-mentioned multiple spatial-channel convolutional layers of the MRI feature extraction module perform multiple spatial convolutions and channel convolutions on the initial convolutional features to obtain multi-level features of the MRI image, including: performing convolutional processing, group normalization processing, and multiple activation processing on the initial convolutional features in the spatial dimension by using the first spatial-channel convolutional layer to obtain the first spatial convolutional features; performing group convolutional processing and pointwise convolutional processing on the first spatial convolutional features in parallel in the channel dimension, and splicing the obtained first group convolutional features and first pointwise convolutional features to obtain the first spatial-channel convolutional features; performing convolutional processing, group normalization processing, and multiple activation processing on the first spatial-channel convolutional features in the spatial dimension by using the second spatial-channel convolutional layer to obtain the second spatial convolutional features; performing group convolutional processing and pointwise convolutional processing on the second spatial convolutional features in parallel in the channel dimension, and splicing the obtained second group convolutional features and second pointwise convolutional features to obtain the second spatial-channel convolutional features; performing convolutional processing, group normalization processing, and multiple activation processing on the second spatial-channel convolutional features in the spatial dimension by using the third spatial-channel convolutional layer to obtain the third spatial convolutional features; performing group convolutional processing and pointwise convolutional processing on the third spatial convolutional features in parallel in the channel dimension, and splicing the obtained third group convolutional features and third pointwise convolutional features to obtain the third spatial-channel convolutional features.

[0097] The working principle of the feature extraction module in the medical image fusion network of the present invention will be further described in detail below through specific embodiments in combination with the attached Figure 3 and 4 drawings.

[0098] Figure 3 It is a schematic structural diagram of a medical image fusion network according to an embodiment of the present invention.

[0099] Figure 4 It is a schematic structural diagram of a spatial and channel convolution unit according to an embodiment of the present invention.

[0100] As Figure 3 shown, the medical image fusion network includes multiple feature extraction modules with correlation constraints, a feature fusion module with an attention guidance mechanism, and a feature reconstruction module. Multimodal images and are input into the medical image fusion network to obtain an image fusion result .

[0101] The correlation-constrained feature extraction module consists of two independent feature extraction branches and a shared feature extraction branch. Each feature extraction branch is composed of a traditional convolutional layer and three spatial and channel convolutions. The convolutional kernel of the first convolutional layer is , padding = 1, stride = 1, which converts the input image into the feature domain. The input channel is 1, and the output channel is 16, and the GeLU activation function is used. The number of input feature channels and output feature channels of each subsequent spatial and channel convolution is the same, and the size remains unchanged. Therefore, the stride of the convolutional layers used in the network is all 1, and residual connections are used to concatenate the input features and output features of each layer on the channel. Therefore, the number of channels of the output features of the four levels are 16, 32, 64, and 128 respectively.

[0102] The structure of each layer of spatial and through convolution is as Figure 4 shown, which contains two parts in series. First, the multimodal features are extracted using the spatial convolution unit, as shown in formula (1):

[0103] (1),

[0104] where GN is Group Normalization, is the Sigmoid activation function. Then, a threshold T = 0.5 is set, and by judging whether the pixel value is greater than the threshold, and are calculated, as shown in formula (2); then multiplied by the original feature value , and then passed through a convolutional layer respectively, with a convolutional kernel size of 1 and the same input and output channels. Finally, the two results are divided into two equal parts, and the two parts of the feature values are cross-summed to obtain the spatial channel output features of three levels , as shown in formula (3):

[0105] (2),

[0106] (3).

[0107] After the spatial unit module, the channel information extraction module is used to focus on extracting inter-channel information. First, the features are divided into two parts through the convolution layer Conv, with the number of channels halved, and then channel information extraction is performed to obtain and , as shown in formulas (4) and (5):

[0108] (4),

[0109] (5),

[0110] where GWC is group convolution, the convolution kernel is , the number of groups group size = 2, PWC is pointwise convolution, that is, convolution, and the output channels of GWC and PWC are both twice the input channels. Concat is a channel concatenation operation. Then, pooling and SoftMax operations are respectively performed on and to obtain self-attention weights, and element-wise sum calculation is performed on the two sets of channel feature information. Finally, channel concatenation is performed to obtain the final output of the spatial channel convolution unit, as shown in formula (6):

[0111] (6).

[0112] According to the embodiments of the present invention, the correlation constraints between the above-mentioned MRI feature extraction module, CT feature extraction module, and common feature extraction module are obtained through the following operations: The medical image fusion network is iteratively trained, and during the iterative training process of the medical image fusion network, the redundancy of the MRI feature extraction module and the CT feature extraction module is reduced, and the correlation between the common feature extraction module, the MRI feature extraction module, and the CT feature extraction module is improved through the correlation constraint loss function in the preset loss function.

[0113] According to an embodiment of the present invention, the iterative training of the medical image fusion network includes: using the medical image fusion network to perform multi-level feature extraction on the preprocessed multi-modal medical image samples in parallel to obtain multi-level features of the multi-modal medical image samples, and using a correlation constraint loss function to perform correlation constraint on the feature extraction process; using the medical image fusion network to perform feature fusion based on a double-layer cross-attention guidance mechanism on the multi-level features of the multi-modal medical image samples at each common level to obtain multi-level fusion features of the multi-modal medical image samples; using the medical image fusion network to splice the multi-level fusion features of the multi-modal medical image samples, and performing image reconstruction on the spliced features of the multi-modal medical image samples to obtain a medical fusion result of the multi-modal medical image samples; using a preset loss function to process the multi-modal medical image samples and the medical image fusion result of the multi-modal medical image samples to obtain a loss value, and updating the parameters of the medical image fusion network based on the loss value; iteratively performing operations of feature extraction, feature fusion, image reconstruction, loss value calculation, and parameter update until a preset training condition is satisfied, and obtaining a trained medical image fusion network.

[0114] The training process of the medical image fusion network and the correlation constraint between multiple feature modules will be further described in detail below through specific embodiments.

[0115] First, obtain a medical image dataset. Collect 180 pairs of registered MRI and CT images with a size of from a publicly available dataset (such as Harvard Whole Brain Atlas) for training. An overlapping cropping strategy with a cropping step size of 20 is adopted, and after horizontal and vertical flipping overlapping cropping, 16380 pairs of images with a length and width of are obtained for training, and 20 pairs of MRI-CT image pairs are used for testing. Preprocess two different-modal MR and CT images respectively, normalize the pixel values to [0,1], and obtain two-modal medical images and after preprocessing; where C, H, and W respectively represent the number of channels, height, and width of the image, the training image is , and the test image is .

[0116] Secondly, construct a loss function as shown in formulas (7) to (10):

[0117] (7),

[0118] (8),

[0119] (9),

[0120] (10),

[0121] where respectively represent the parameters to be optimized for the feature extraction module, feature fusion module, and feature reconstruction module, are respectively the mean and variance of is the covariance of. SSIM_L represents the image similarity loss function, MSE_L represents the mean squared error loss function, represents the feature correlation loss function, and the hyperparameters α and β are used to control the balance between different loss functions.

[0122] Finally, use the Adam optimizer to train the image fusion network model, calculate the loss function and backpropagate it into the network for optimization, and stop training when the loss function converges, so as to obtain the optimal image fusion model. In this embodiment, it is carried out using the PyTorch framework on an NVIDIA GeForce RTX3060 GPU. During the training process, the learning rate is 1e-4, the batch size batchsize is 16, the number of epochs is set to 4, and the hyperparameters is 1, is 5.

[0123] In addition, use the trained medical image fusion model to fuse 20 MRI-CT images. The fusion effect of the model is evaluated by combining qualitative evaluation and quantitative evaluation.

[0124] According to the embodiments of the present invention, the above-mentioned feature fusion module with an attention guidance mechanism in the medical image fusion network performs double-layer cross-attention feature fusion on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel splicing image at each common level, and obtains multi-level fusion features with different numbers of channels, including: using the multi-level features of the channel splicing image as the common features to generate the multi-level query vectors in the attention guidance mechanism; using the multi-level features of the MRI image as the unique features to generate the MRI multi-level key vectors and MRI multi-level value vectors in the attention guidance mechanism; using the multi-level features of the CT image as the unique features to generate the CT multi-level key vectors and CT multi-level value vectors in the attention guidance mechanism; at each common level, using the common feature fusion module of the medical image fusion network to perform cross-attention feature fusion guided by common features on the multi-level query vectors, MRI multi-level key vectors, MRI multi-level value vectors, CT multi-level key vectors, and CT multi-level value vectors, and obtaining multi-level common fusion features with different numbers of channels.

[0125] The above-mentioned feature fusion module with an attention guidance mechanism in the medical image fusion network performs double-layer cross-attention feature fusion on the multi-level features of MRI images, the multi-level features of CT images, and the multi-level features of channel concatenation images at each common level, obtaining multi-level fusion features with different numbers of channels. By using the channel concatenation features as the common query vectors to guide the interaction of MRI / CT specific key-value vectors, the dynamic collaboration between the common features (such as organ contours) and specific features (CT bones / MRI soft tissues) between modalities is realized, significantly improving the utilization rate of complementary information. At the same time, performing cross-attention on the same channel dimension can not only strengthen the modality-specific features (such as tumor regions) through the MRI / CT key-value vectors, but also maintain the anatomical structure consistency through the common query vectors, achieving the dual guidance of "global alignment + local optimization". In addition, generating attention weights for the multi-level features respectively enables the independent optimization and fusion of shallow details (edge textures) and deep semantics (organ shapes), avoiding interference between features of different scales and improving the resolution and structural coherence of the fused images. Moreover, the cross-attention calculation with a fixed number of channels (compared with dynamic channel adjustment) reduces a large amount of matrix operation volume, and at the same time, through the differential output of the number of channels, the computational load of the subsequent reconstruction module is balanced.

[0126] According to an embodiment of the present invention, the above-mentioned feature fusion module with an attention guidance mechanism in the medical image fusion network performs double-layer cross-attention feature fusion on the multi-level features of MRI images, the multi-level features of CT images, and the multi-level features of channel concatenation images at each common level, obtaining multi-level fusion features with different numbers of channels further includes: using the multi-level features of the channel concatenation images as common features to generate multi-level key vectors and multi-level value vectors in the attention guidance mechanism; using the multi-level features of the MRI images as specific features to generate MRI multi-level query vectors in the attention guidance mechanism; using the multi-level features of the CT images as specific features to generate CT multi-level query vectors in the attention guidance mechanism; at each common level, using the specific feature fusion module of the medical image fusion network to perform cross-attention feature fusion with specific feature guidance on the multi-level key vectors, multi-level value vectors, MRI multi-level query vectors, and CT multi-level query vectors, obtaining multi-level specific fusion features with different numbers of channels; performing element-wise operations on the multi-level common fusion features with different numbers of channels and the multi-level specific fusion features with different numbers of channels on the same number of channels, obtaining multi-level fusion features with different numbers of channels.

[0127] The following will further elaborate on the above-mentioned multi-modal image fusion process provided by the present invention through specific embodiments and in combination with the attached Figure 5 drawings.

[0128] Figure 5Schematic diagram of the multi-modal feature fusion process guided by double-layer cross attention according to an embodiment of the present invention.

[0129] To achieve the interactive fusion of four different levels of multi-modal features, a double-layer cross-attention fusion mechanism is adopted, as Figure 5 shown, and two sub-modules are processed in parallel for feature fusion. The two parallel cross-attention modules achieve the interactive fusion between the unique features and the shared features. One is the unique feature fusion guided by the shared feature, and the other is the shared feature fusion guided by the unique feature. For the two types of modal unique features and modal shared features at four levels, the following operations are performed, and finally the features at four levels after fusion are obtained , and the number of feature channels is 16, 32, 64, and 128 respectively.

[0130] First, in the first-layer attention fusion process, the cross-modal shared features are used to generate the query (Q) of the attention mechanism, while the modality-specific features are used to generate the key (K) and value (V), so that the unique feature information within the modality can be effectively integrated through the information provided by the attention mechanism, as shown in formula (11):

[0131] (11),

[0132] where , , , and represent learnable weight matrices, represents the query vector converted from the shared feature , and represent the key and value vectors converted from the first modality-specific feature , and represent the key and value vectors converted from the second modality-specific feature . Then, is input into the cross-attention layer to obtain . At the same time, is input into the cross-attention layer to obtain . The definition of the ATTN module is shown in formula (12):

[0133] (12).

[0134] Secondly, correspondingly, in the second-layer attention fusion process, the query (Q) is generated from the modality-specific features, and the key (K) and value (V) are generated from the shared features, so as to promote the fusion of shared information, as shown in formula (13):

[0135] (13),

[0136] Among them, , , , are learnable weight matrices, and represent query vectors transformed from modality-specific features and . The and represent key and value vectors transformed from the shared feature . Input into the attention layer to obtain . At the same time, input into the attention layer to obtain .

[0137] Finally, perform an element-wise addition operation on each level of the obtained to obtain the final fusion result at each level, and the four-level fusion feature .

[0138] During the image reconstruction process, after performing double-layer cross-attention interaction on the four levels, the four-level fusion features obtained are concatenated along the channels and sent to the image reconstruction module to generate the final fused image. The image reconstruction module consists of a transposed convolutional layer and an activation function to achieve feature dimensionality reduction. First, perform a convolution of with padding = 1 to reduce the dimensionality of the concatenated features with 240 channels to 60 channels, then pass through the GeLu activation function, and then send it to a convolutional layer with an input channel of 60 and an output channel of 30. Then, enhance the nonlinearity of the network through a layer of ReLU activation function. Finally, obtain the final fusion result through a convolutional layer with an input channel of 30 and an output channel of 1 and a Sigmoid activation function, as shown in Formulas (14) and (15):

[0139] (14),

[0140] (15).

[0141] The above-mentioned medical image fusion method based on correlation constraint and attention guidance provided by the present invention introduces linear and non-linear correlation loss functions in the feature extraction module to achieve more sufficient constraints on the multi-modal feature extraction branch, helping to retain important information and reduce redundancy, and realizing a more balanced extraction of shared information between modalities and specific information within modalities. The experimental results on the publicly available medical image dataset show that the image fusion effect of the method proposed by the present invention is better than that of multiple existing image fusion networks. At the same time, the present invention proposes a double-layer cross-attention fusion module, which respectively focuses on the interactive guidance fusion of specific information and shared information to achieve more refined fusion and reduce the loss of important information. The double-layer cross-attention interactive feature fusion module consists of two parallel cross-attention fusion modules. One is the unique feature fusion guided by shared features, and the other is the shared feature fusion guided by unique features, which promotes the interaction between common features and unique features during fusion and further improves the effect of image fusion.

[0142] Figure 6 It is a schematic diagram of the multi-modal image processing process according to an embodiment of the present invention.

[0143] As Figure 6 shown, two modalities of medical images are input, and then the two modalities of medical images are concatenated in channels. The channel-concatenated image is input into the modality-common branch for processing to obtain common features, and the two modalities of medical images input are input into the modality-unique branch to obtain unique features; then, feature fusion of two types (i.e., double-layer cross-attention) is performed; finally, the fused features are added and reconstructed to obtain the multi-modal medical fusion image.

[0144] The advantages of the method provided by the present invention will be described below through comparative experiments with other multi-modal medical image fusion methods and in combination with the accompanying drawings.

[0145] Figure 7 It is a comparative effect diagram of the multi-type multi-modal medical image fusion method according to an embodiment of the present invention.

[0146] The qualitative results of this experiment are as Figure 7 shown. The final performance of the methods used in the comparative experiment is evaluated by four common image quality indicators: information theory-based normalized mutual information ( ), image feature-based normalized gradient index , structural similarity (SSIM) with the source image, and visual quality of the image

[0147] . The value ranges of these indicators are between 0 and 1, and the closer to 1, the better the fusion effect.

[0148] The comparison methods used in the comparative experiments are U2Fusion (A Unified Unsupervised Image Fusion Network), EMFusion (An Unsupervised Enhanced Medical Image Fusion Network), CDDFusion (Correlation-driven Dual-branch Feature Decomposition for Multi-modality Image Fusion), and EMMA (Equivariant Multi-Modality Image Fusion). The fusion effect indicators of all methods are shown in Table 1.

[0149] Table 1 Average fusion results of different methods on the MRI-CT dataset

[0150]

[0151] The method provided by the present invention has achieved the optimal results in these four indicators, which indicates that the method proposed by the present invention has achieved the best image fusion effect and improved the quality of medical image fusion. Figure 7 The results show that the fused images of the invention retain more MRI texture details and CT bone density information. In the first example, compared with other methods such as U2Fusion, EMFusion, and EMMA, the results of the present invention maintain a brighter pixel intensity, indicating that high-intensity information is better retained. In the CDDFusion method, the structural details of magnetic resonance imaging are either lost or changed, as shown in the second and third comparative examples. The present invention provides a new solution for optimizing multi-modal medical image fusion technology, which is beneficial to subsequent analysis and applications based on multi-modal medical image fusion.

[0152] Figure 8 It is a schematic structural diagram of a medical image fusion system based on correlation constraint and attention guidance according to an embodiment of the present invention.

[0153] As Figure 8 shown, the above-mentioned medical image fusion system 800 based on correlation constraint and attention guidance includes a channel splicing image acquisition module 810, a multi-level feature extraction module 820, a multi-channel feature fusion module 830, and a medical image fusion reconstruction module 840.

[0154] The channel stitching image acquisition module 810 is configured to stitch the preprocessed MRI image and the preprocessed CT image on the channels to obtain a channel stitched image; in one embodiment, the channel stitching image acquisition module 810 can be used to perform the operation S210 described above, which will not be elaborated here.

[0155] The multi-level feature extraction module 820 is configured to perform multi-level feature extraction on the channel stitched image, the preprocessed MRI image, and the preprocessed CT image by using the feature extraction module with correlation constraints in the medical image fusion network to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel stitched image; in one embodiment, the multi-level feature extraction module 820 can be used to perform the operation S220 described above, which will not be elaborated here.

[0156] The multi-channel feature fusion module 830 is configured to perform feature fusion of double-layer cross-attention on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel stitched image at each common level by using the feature fusion module with an attention guidance mechanism in the medical image fusion network to obtain multi-level fusion features with different numbers of channels; in one embodiment, the multi-channel feature fusion module 830 can be used to perform the operation S230 described above, which will not be elaborated here.

[0157] The medical image fusion and reconstruction module 840 is configured to stitch the multi-level fusion features with different numbers of channels on the channels by using the feature reconstruction module of the medical image fusion network to obtain a channel stitched fusion feature, and perform multi-modal medical image reconstruction on the channel stitched fusion feature to obtain a medical image fusion result. In one embodiment, the medical image fusion and reconstruction module 840 can be used to perform the operation S240 described above, which will not be elaborated here.

[0158] According to an embodiment of the present invention, any plurality of modules among the channel splicing image acquisition module 810, the multi-level feature extraction module 820, the multi-channel feature fusion module 830, and the medical image fusion and reconstruction module 840 may be combined and implemented in one module, or any one of them may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the channel splicing image acquisition module 810, the multi-level feature extraction module 820, the multi-channel feature fusion module 830, and the medical image fusion and reconstruction module 840 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or may be implemented by any other reasonable means such as hardware or firmware through circuit integration or packaging, or may be implemented in any one of the three implementation manners of software, hardware, and firmware or in an appropriate combination of any several of them. Alternatively, at least one of the channel splicing image acquisition module 810, the multi-level feature extraction module 820, the multi-channel feature fusion module 830, and the medical image fusion and reconstruction module 840 may be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0159] Figure 9 is a block diagram of an electronic device suitable for implementing a medical image fusion method based on correlation constraint and attention guidance according to an embodiment of the present invention.

[0160] As Figure 9 shown, the electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 901 may also include on-board memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0161] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to the embodiments of the present invention by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to the embodiments of the present invention by executing the programs stored in the one or more memories.

[0162] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, and the input / output (I / O) interface 905 is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read from it can be installed into the storage section 908 as needed.

[0163] The present invention also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0164] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include one or more memories other than the ROM 902 and / or RAM 903 and / or ROM 902 and RAM 903 described above.

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0166] Those skilled in the art can understand that the features described in various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0167] The above describes the embodiments of the present invention. However, these embodiments are only for illustrative purposes and not for limiting the scope of the present invention. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all these substitutions and modifications should fall within the scope of the present invention.

Claims

1. A medical image fusion method based on correlation constraint and attention guidance, characterized in that The method includes: Performing channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain a channel-stitched image; Using a feature extraction module with correlation constraints in a medical image fusion network to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image; Using a feature fusion module with an attention guidance mechanism in the medical image fusion network to perform feature fusion of double-layer cross-attention on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-stitched image at each common level to obtain multi-level fusion features with different numbers of channels; Using the feature reconstruction module of the medical image fusion network to perform channel stitching on multiple fusion features with different numbers of channels to obtain a channel-stitched fusion feature, and performing multi-modal medical image reconstruction on the channel-stitched fusion feature to obtain a medical image fusion result.

2. The method according to claim 1, wherein Performing channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain a channel-stitched image includes: Performing pixel value normalization processing on the MRI image and the CT image respectively to obtain the preprocessed MRI image and the preprocessed CT image; Performing channel stitching on the preprocessed MRI image and the preprocessed CT image to obtain the channel-stitched image.

3. The method according to claim 1, characterized in that, Using a feature extraction module with correlation constraints in a medical image fusion network to perform multi-level feature extraction on the channel-stitched image, the preprocessed MRI image, and the preprocessed CT image to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel-stitched image includes: Using the MRI feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed MRI image to obtain the multi-level features of the MRI image; Using the CT feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed CT image to obtain the multi-level features of the CT image; Using the common feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed channel-stitched image to obtain the multi-level features of the channel-stitched image, where there is a correlation constraint between the MRI feature extraction module, the CT feature extraction module, and the common feature extraction module.

4. The method according to claim 3, characterized in that, Using the MRI feature extraction module of the medical image fusion network to perform convolution processing and multiple spatial-channel convolution processing on the preprocessed MRI image to obtain the multi-level features of the MRI image includes: Using the initial convolution layer of the MRI feature extraction module to transform the preprocessed MRI image into the feature domain to obtain initial convolution features; Performing multiple spatial convolutions and channel convolutions on the initial convolutional features using multiple spatial channel convolutional layers of the MRI feature extraction module to obtain multi-level spatial channel convolutional features with different numbers of channels, where the multi-level features of the MRI image include the initial convolutional features and multiple spatial channel convolutional features with different numbers of channels.

5. The method according to claim 4, wherein Performing multiple spatial convolutions and channel convolutions on the initial convolutional features using multiple spatial channel convolutional layers of the MRI feature extraction module to obtain the multi-level features of the MRI image, including: Performing convolutional processing, group normalization processing, and multiple activation processing on the initial convolutional features in the spatial dimension using the first spatial channel convolutional layer to obtain the first spatial convolutional features; Performing group convolutional processing and pointwise convolutional processing on the first spatial convolutional features in the channel dimension in parallel, and concatenating the obtained first group convolutional features and first pointwise convolutional features to obtain the first spatial channel convolutional features; Performing convolutional processing, group normalization processing, and multiple activation processing on the first spatial channel convolutional features in the spatial dimension using the second spatial channel convolutional layer to obtain the second spatial convolutional features; Performing group convolutional processing and pointwise convolutional processing on the second spatial convolutional features in the channel dimension in parallel, and concatenating the obtained second group convolutional features and second pointwise convolutional features to obtain the second spatial channel convolutional features; Performing convolutional processing, group normalization processing, and multiple activation processing on the second spatial channel convolutional features in the spatial dimension using the third spatial channel convolutional layer to obtain the third spatial convolutional features; Performing group convolutional processing and pointwise convolutional processing on the third spatial convolutional features in the channel dimension in parallel, and concatenating the obtained third group convolutional features and third pointwise convolutional features to obtain the third spatial channel convolutional features.

6. The method according to claim 3, wherein The correlation constraint between the MRI feature extraction module, the CT feature extraction module, and the common feature extraction module is obtained through the following operations: Performing iterative training on the medical image fusion network, and during the iterative training process of the medical image fusion network, reducing the redundancy of the MRI feature extraction module and the CT feature extraction module and increasing the correlation between the common feature extraction module, the MRI feature extraction module, and the CT feature extraction module through the correlation constraint loss function in the preset loss function.

7. The method according to claim 6, wherein Performing iterative training on the medical image fusion network includes: Performing multi-level feature extraction on the preprocessed multi-modal medical image samples in parallel using the medical image fusion network to obtain the multi-level features of the multi-modal medical image samples, and using the correlation constraint loss function to perform correlation constraints on the feature extraction process; Performing feature fusion based on the double-layer cross-attention guidance mechanism on the multi-level features of the multi-modal medical image samples at each common level using the medical image fusion network to obtain the multi-level fusion features of the multi-modal medical image samples; The multi-level fusion features of the multi-modal medical image samples are spliced using the medical image fusion network, and image reconstruction is performed on the spliced features of the multi-modal medical image samples to obtain the medical fusion result of the multi-modal medical image samples; The preset loss function is used to process the multi-modal medical image samples and the medical image fusion result of the multi-modal medical image samples to obtain a loss value, and the parameters of the medical image fusion network are updated based on the loss value; The operations of feature extraction, feature fusion, image reconstruction, loss value calculation, and parameter update are iteratively performed until the preset training conditions are met, and the trained medical image fusion network is obtained.

8. The method according to claim 1, wherein Using the feature fusion module with an attention guidance mechanism in the medical image fusion network, the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel-spliced image are subjected to feature fusion of double-layer cross-attention at each common level to obtain multi-level fusion features with different numbers of channels, including: The multi-level features of the channel-spliced image are used as common features to generate multi-level query vectors in the attention guidance mechanism; The multi-level features of the MRI image are used as unique features to generate MRI multi-level key vectors and MRI multi-level value vectors in the attention guidance mechanism; The multi-level features of the CT image are used as unique features to generate CT multi-level key vectors and CT multi-level value vectors in the attention guidance mechanism; At each common level, the common feature fusion module of the medical image fusion network is used to perform cross-attention feature fusion guided by common features on the multi-level query vectors, the MRI multi-level key vectors, the MRI multi-level value vectors, the CT multi-level key vectors, and the CT multi-level value vectors to obtain multi-level common fusion features with different numbers of channels.

9. The method according to claim 8, characterized in that It also includes: The multi-level features of the channel-spliced image are used as common features to generate multi-level key vectors and multi-level value vectors in the attention guidance mechanism; The multi-level features of the MRI image are used as unique features to generate MRI multi-level query vectors in the attention guidance mechanism; The multi-level features of the CT image are used as unique features to generate CT multi-level query vectors in the attention guidance mechanism; At each common level, the unique feature fusion module of the medical image fusion network is used to perform cross-attention feature fusion guided by unique features on the multi-level key vectors, the multi-level value vectors, the MRI multi-level query vectors, and the CT multi-level query vectors to obtain multi-level unique fusion features with different numbers of channels; Element-wise operations are performed on multiple common fusion features with different numbers of channels and multiple unique fusion features with different numbers of channels on the same number of channels to obtain multiple fusion features with different numbers of channels.

10. A medical image fusion system based on correlation constraint and attention guidance, characterized in that, The system includes: A channel-spliced image acquisition module for splicing the preprocessed MRI image and the preprocessed CT image on the channels to obtain a channel-spliced image; A multi-level feature extraction module, which is used to perform multi-level feature extraction on the channel concatenated image, the preprocessed MRI image, and the preprocessed CT image by using the feature extraction module with correlation constraints in the medical image fusion network, so as to obtain multi-level features of the MRI image, multi-level features of the CT image, and multi-level features of the channel concatenated image; A multi-channel feature fusion module, which is used to perform feature fusion of double-layer cross-attention on the multi-level features of the MRI image, the multi-level features of the CT image, and the multi-level features of the channel concatenated image at each common level by using the feature fusion module with an attention guidance mechanism in the medical image fusion network, so as to obtain multi-level fusion features with different numbers of channels; A medical image fusion and reconstruction module, which is used to splice the multiple fusion features with different numbers of channels on the channels by using the feature reconstruction module of the medical image fusion network to obtain a channel concatenated fusion feature, and perform multi-modal medical image reconstruction on the channel concatenated fusion feature to obtain a medical image fusion result.

Images