Medical image fusion method and system based on big kernel attention mechanism

Abstract

The invention provides a medical image fusion method and system based on a big kernel attention mechanism, and belongs to the technical field of medical image fusion. A big kernel attention mechanism and frequency domain transformation processing are introduced, and a multi-scale convolution operation, channel attention and a space attention mechanism are combined, so that the fusion image achieves a higher balance in the retention of global information and local details. Meanwhile, through separation processing of amplitude and phase information in a frequency domain, accurate fusion of image details and structures is realized, and common artifact and fuzzy problems in a traditional method are avoided. Besides, a densely connected decoder module is designed, the detail retention capability during image reconstruction is enhanced, and processing of different modal image features is more intelligent and accurate through a self-adaptive feature fusion mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image fusion, and in particular relates to a medical image fusion method and system based on a large-core attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of modern medical imaging technology, multimodal medical imaging technologies such as MRI (magnetic resonance imaging) and PET (positron emission tomography) are widely used in clinical diagnosis. MRI can provide high-resolution soft tissue anatomical information, while PET can reflect cell metabolic activities. By fusing these two images, doctors can observe the patient's lesions more comprehensively, thereby improving the accuracy of diagnosis and treatment effects. However, how to efficiently and accurately fuse multimodal images has become an important research direction in the field of medical image processing.

[0004] In the prior art, common image fusion methods mainly include fusion methods based on pixel level, feature level and decision level. Pixel-level fusion methods directly process and synthesize the pixel values ​​of the source image. Typical algorithms include wavelet transform, pyramid transform, etc. However, these methods are prone to lose important structural information or introduce artifacts, resulting in low quality of fused images. Feature-level fusion methods fuse by extracting the features of the source images. Although the quality of the fused image is improved, the feature selection and processing process is complex and often requires a lot of computing resources. Decision-level fusion methods use the classification or recognition results of the image to make high-level decisions, but this method usually needs to rely on prior knowledge and is difficult to adapt to the needs of multimodal image fusion in complex scenes.

[0005] In recent years, deep learning technology has been introduced into the field of image fusion and has made significant progress. Through deep learning models such as convolutional neural networks (CNNs), image fusion methods can more effectively learn the deep features of images, thereby achieving more intelligent image fusion processing. Some studies have proposed the use of self-attention mechanisms to enhance feature selection and representation capabilities in the fusion process, but existing methods often ignore the full use of multi-scale information in the feature extraction process. Multi-scale features are of great significance in medical image fusion because images of different modalities may contain key information of different scales.

[0006] In addition, the attention mechanism in the existing technology usually uses a smaller kernel size for convolution operations. Although it can improve the local perception ability of the model, it cannot fully capture the global context information in medical images. Therefore, how to design a more effective attention mechanism, especially for capturing large-scale context, becomes the key to improving image fusion effects. Summary of the invention

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a medical image fusion method and system based on a large core attention mechanism, introduces a large core attention mechanism and frequency domain transformation processing, and combines multi-scale convolution operations, channel attention and spatial attention mechanisms, so that the fused image can achieve a higher balance in the retention of global information and local details. At the same time, by separating the amplitude and phase information in the frequency domain, the precise fusion of image details and structures is achieved, avoiding the common artifacts and blurring problems in traditional methods. In addition, the present invention also designs a densely connected decoder module to enhance the detail retention capability during image reconstruction, and through an adaptive feature fusion mechanism, it is more intelligent and accurate in processing features of different modal images.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides a medical image fusion method based on a large kernel attention mechanism; A medical image fusion method based on large kernel attention mechanism, including: Acquire multimodal medical images, and preprocess the acquired multimodal medical images; The preprocessed multimodal medical image is input into the dual-branch encoder and preliminary feature extraction is performed through the multi-scale convolutional layer; The large-core attention mechanism is used to perform group normalization, projection, and segmentation on the initially extracted features to obtain multiple features after segmentation. The segmented features are respectively input into large-core attention submodules of different scales and corresponding spatial feature extraction convolution layers to obtain global and local features of the image: the global and local features are concatenated and projected to the original dimension; Utilize spectrum dynamic aggregation to perform Fourier transform on the multi-scale features projected to the original dimension, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; Multi-scale convolution, channel attention and spatial attention mechanisms are used to process the features after Fourier transformation to achieve adaptive fusion of features of different modalities and regions and adaptive weight adjustment; The fused features are processed using a densely connected decoder to obtain the final fused image representation.

[0009] As a further technical solution, the multimodal medical image includes MRI magnetic resonance imaging and PET positron emission tomography images.

[0010] As a further technical solution, the process of preprocessing the acquired multimodal medical image is: adjusting the input image to a standard size of 128×128, and ensuring that the pixel value range of the multimodal medical image is consistent through a normalization operation.

[0011] As a further technical solution, the process of inputting the preprocessed multimodal medical image into the dual-branch encoder and performing preliminary feature extraction through the multi-scale convolutional layer is as follows: The preprocessed MRI and PET images are respectively input into their respective branch encoders, and feature extraction is performed through a multi-scale convolution layer; the multi-scale convolution uses convolution kernels of different sizes to capture feature information of different scales in the medical image; ; ; in, and Represents the multi-scale convolutional extractor for the MRI and PET branches; the output and It is the preliminary feature representation of different modalities; and They are the preprocessed MRI and PET images respectively.

[0012] As a further technical solution, the segmented multiple medical image features are respectively input into large-core attention submodules of different scales and corresponding spatial feature extraction convolution layers to obtain the global and local features of the image:

[0013] In the formula, , , It is the global feature map obtained after being processed by large-core attention submodules of different scales; , , It is the local feature map obtained after processing by convolutional layers of spatial feature extraction of different scales; LKA3, LKA5, and LKA7 are large-core attention submodules of different scales; X3, X5, and X7 are the corresponding 3x3, 5x5, and 7x7 spatial feature extraction convolutional layers; , , are multiple medical image features after segmentation; The process of concatenating the global and local features and projecting them to the original dimension is: multiplying the large kernel convolution result and the spatial feature extraction result to enhance the representation capability of multi-scale features:

[0014] In the formula , It is the feature representation after multi-scale feature fusion, which contains the combined information of global and local features; The concatenated multi-scale features are projected back to the original dimension through the projection convolution layer to ensure that the feature size is consistent with the input: ; ; In the formula, , It is the final feature representation obtained after the last projection layer, and its dimension is consistent with the input feature; It is the projection convolution layer.

[0015] As a further technical solution, the process of using spectrum dynamic aggregation to perform Fourier transform on the multi-scale features projected to the original dimension and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain is as follows: The spectral dynamic aggregation module performs Fourier transform on the multi-scale features projected to the original dimension, transforming them from the spatial domain Convert to frequency domain , respectively obtain the amplitude and phase information of the feature; ; ; in, represents the Fourier transform operation, and Respectively represent the representation of MRI and PET image features in the frequency domain; after Fourier transform, the features are decomposed into amplitude information and phase information ; ; ; in, and Represent the amplitude information of MRI and PET images respectively, and is the phase information, is the Euler representation on the complex plane, i is the imaginary unit, is the phase angle.

[0016] As a further technical solution, a medical image fusion method based on a large kernel attention mechanism also includes a loss function design.

[0017] A second aspect of the present invention provides a medical image fusion system based on a large kernel attention mechanism.

[0018] A medical image fusion system based on large kernel attention mechanism, including: The multimodal medical image acquisition module is configured to: acquire the multimodal medical image and preprocess the acquired multimodal medical image; The large core attention mechanism processing module is configured to: input the preprocessed multimodal medical image into the dual-branch encoder, perform preliminary feature extraction through the multi-scale convolution layer; use the large core attention mechanism to perform group normalization, projection and segmentation processing on the preliminary extracted features to obtain multiple features after segmentation; The multi-scale large-kernel convolution and spatial feature extraction module is configured to: input the segmented multiple features into large-kernel attention sub-modules of different scales and corresponding spatial feature extraction convolution layers respectively to obtain global and local features of the image; and concatenate the global and local features and then project them to the original dimension; The spectrum dynamic aggregation module is configured to: perform Fourier transform on the multi-scale features projected to the original dimension using the spectrum dynamic aggregation module, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; The feature fusion and adaptive weight adjustment module is configured to: process the features after Fourier transformation by using multi-scale convolution, channel attention and spatial attention mechanisms, and realize adaptive fusion of features of different modalities and different regions and adaptive weight adjustment; The medical image representation module is configured to: process the fused features using a densely connected decoder to obtain a final fused medical image representation.

[0019] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a medical image fusion method based on a large core attention mechanism as described in the first aspect of the present invention.

[0020] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a medical image fusion method based on a large core attention mechanism as described in the first aspect of the present invention are implemented.

[0021] One or more of the above technical solutions have the following beneficial effects: The present invention combines multi-scale feature extraction, large kernel attention mechanism and spectrum dynamic aggregation to efficiently process the features of MRI and PET images in the frequency domain. Compared with traditional methods, the present invention can not only capture the local details and global structure of the image, but also effectively improve the modeling ability of long-distance dependent features through large kernel convolution. At the same time, the proposed global-local adaptive fusion strategy can dynamically adjust the feature weights according to the importance of different modalities and regions, ensuring the accurate fusion of information.

[0022] In addition, the present invention utilizes a dual-branch encoder to extract preprocessed MRI and PET images, which is beneficial for extracting and fusing multimodal features of images; the decoder fully utilizes multi-scale features through a densely connected structure, further improving the quality of image reconstruction and reducing information loss and artifacts. Unlike the traditional symmetric codec structure, the asymmetric codec architecture provided by the present invention is more suitable for the characteristics of multimodal medical image fusion, and ultimately outputs a fused image with better quality. The phase and amplitude consistency in the frequency domain is further enhanced by introducing phase and amplitude loss, thereby ensuring the high quality of the fused image. In practical applications, the present invention can effectively improve the accuracy of medical image fusion, ensure richer detail presentation and more accurate diagnostic information, and provide strong technical support for medical image analysis.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a flow chart of the method of the first embodiment.

[0026] Figure 2 Schematic diagram of the structure of the large core attention mechanism module in the first embodiment.

[0027] Figure 3 It is a schematic diagram of the spectrum dynamic aggregation process in the first embodiment.

[0028] Figure 4a is the MRI image in the first embodiment; Figure 4b is a PET image in the first embodiment; Figure 4c is the fused image.

[0029] Figure 5 It is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0032] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0033] MRI and PET images provide anatomical structure and functional metabolic information respectively. The images of these two modalities have significant differences in feature distribution, spatial resolution, etc. Existing image fusion methods, such as pixel-level and feature-level fusion, usually cannot effectively capture the complementary information between the two modalities; at the same time, most medical image fusion methods are limited to processing in the spatial domain and fail to fully utilize frequency domain information.

[0034] In medical images, it is necessary to preserve global anatomical information (such as the outline and morphology of organs) while also carefully depicting the details of local lesion areas (such as the morphological characteristics of lesions). Existing small kernel convolution and local attention mechanisms perform well in processing local features, but have limited ability to capture global contextual information and are prone to losing some key global structural information.

[0035] Based on the above problems, the present invention proposes a medical image fusion method and system based on a large-core attention mechanism, which will be described in detail below in conjunction with specific embodiments.

[0036] Embodiment 1 This embodiment discloses a medical image fusion method based on a large kernel attention mechanism; like Figure 1 As shown, a medical image fusion method based on a large kernel attention mechanism includes: Step S1, obtaining a multimodal medical image and preprocessing the obtained multimodal medical image; wherein, in this embodiment, the multimodal medical image is an MRI image and a PET image, and the process of preprocessing the MRI image and the PET image includes: standardization of the image size, that is, adjusting the input image to a standard size of 128×128, and ensuring that the pixel value range of each modality medical image is consistent through a normalization operation. This step helps to eliminate the scale difference between different image modalities and lays the foundation for subsequent feature extraction.

[0037] In the data flow process, the first operations to be completed are as follows: ; ; in, and They are the preprocessed MRI images and PET images respectively.

[0038] Step S2, inputting the preprocessed multimodal medical image into a dual-branch encoder, performing preliminary feature extraction through a multi-scale convolutional layer; performing group normalization, projection and segmentation processing on the preliminary extracted features using a large core attention mechanism to obtain multiple features after segmentation; Specific, combined Figure 2 In step S2, the preprocessed MRI and PET images are input into their respective encoders, and feature extraction is first performed through a multi-scale convolution layer. In this embodiment, the encoder adopts a dual-branch structure, each branch contains multiple convolution blocks, and each convolution block consists of a standard convolution, batch normalization, activation function, and a large core attention module. The dual-branch encoder uses a skip connection method to connect features layer by layer to enhance feature expression. Finally, the dual-branch features are adaptively fused through a fusion module.

[0039] In the dual-branch encoder, the multi-scale convolutional layer uses convolution kernels of different sizes (such as 3×3, 5×5, 7×7) to capture feature information of different scales in the image.

[0040] ; ; in, and Represents the multi-scale convolution extractor of the MRI and PET branches, and the output and It is the preliminary feature representation of different modalities.

[0041] The initial extracted features and Perform group normalization to reduce the feature distribution differences between channels and improve training stability.

[0042] ; ; In the formula, Normalize for the group.

[0043] The normalized features are passed through the projection convolution layer Projected into a higher dimensional space. It is a 1×1 convolution layer, which is used to expand the dimension of features so that subsequent multi-scale convolution operations can operate in a richer feature space. The specific projection operation is: ; ; , is the feature map after projection; the number of channels of the feature after projection is expanded from C to 2C.

[0044] The projected features are divided into two parts, denoted as and ,in, Represents the part used to maintain the residual connection after feature segmentation; It will be further divided into three parts , and ;

[0045] Step S3, inputting the segmented multiple features into large-core attention submodules of different scales and corresponding spatial feature extraction convolution layers respectively to obtain global and local features of the image: concatenating the global and local features and projecting them to the original dimension; After the division , and They are input into large-core attention submodules (LKA3, LKA5, LKA7) of different scales and corresponding spatial feature extraction convolution layers (X3, X5, X7) to process global and local features respectively:

[0046] In the formula, , , It is the global feature map obtained after being processed by large-core attention submodules of different scales; , , It is the local feature map obtained after processing by convolutional layers of spatial feature extraction of different scales; LKA3, LKA5, and LKA7 are large-core attention submodules of different scales. Each LKA module contains a combination of deep convolution and dilated convolution to expand the receptive field; X3, X5, and X7 are corresponding 3x3, 5x5, and 7x7 spatial feature extraction convolutional layers; , , are multiple medical image features after segmentation.

[0047] Multiply the large kernel convolution result and the spatial feature extraction result to enhance the representation capability of multi-scale features:

[0048] In the formula , It is the feature representation after multi-scale feature fusion, which contains the combined information of global and local features; The concatenated multi-scale features are passed through the projection convolution layer Project back to the original dimension to ensure that the feature size is consistent with the input: and ; In the formula, , They respectively represent the final feature representations of MRI and PET images after being processed by the projection convolution layer. These feature dimensions are consistent with the input, preparing for subsequent frequency domain processing.

[0049] Step S4, using spectrum dynamic aggregation to perform Fourier transform on the multi-scale features projected to the original dimension, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; Combination Figure 3 , first, the features extracted in the dual-branch encoder (such as MRI and PET encoder) and After multi-scale large kernel convolution processing, it is input into the spectrum dynamic aggregation module. First, the spectrum dynamic aggregation module performs Fourier transform on the input features to transform them from the spatial domain Convert to frequency domain , and obtain the amplitude and phase information of the feature respectively.

[0050] ; in, represents the Fourier transform operation, and Respectively represent the representation of MRI and PET image features in the frequency domain. After Fourier transform, the features are decomposed into amplitude information and phase information : and ; in, and Respectively represent the amplitude information of MRI and PET images, and is the phase information, is the Euler representation on the complex plane, i is the imaginary unit, is the phase angle. This complex number representation can also contain amplitude information and phase information , which helps to complete the results and detail features of the preserved image in the frequency domain.

[0051] Amplitude information reflects the global structure and brightness distribution in the image. Therefore, the amplitude processing part in the spectrum dynamic aggregation module dynamically adjusts the amplitude information of MRI and PET features. Specifically, convolutional layers and nonlinear activation functions (such as Leaky ReLU) are used to enhance the amplitude information to ensure that the global structural information is preserved and strengthened. The processing of amplitude information can be expressed as: ; ; and It is the amplitude information extracted from the MRI and PET features after Fourier transformation. represents the convolutional layer used to enhance the amplitude information, is a nonlinear activation function. The processed amplitude information is passed through the second convolutional layer Further adjustments are made to ensure that it can adapt to subsequent feature fusion tasks.

[0052] Finally, the processed amplitude information and are reassembled back into frequency domain features.

[0053] Secondly, in the process of processing the amplitude and phase information of multi-scale features and realizing dynamic aggregation in the frequency domain, the phase information contains the edge, texture and detail information in the image, which is crucial to the visual effect of the image. The FSDA module processes the phase information to enhance the edges and details of the image. In the specific implementation, convolution operations and attention mechanisms are used to dynamically adjust the phase information. The processing process of phase information is as follows: Phase information mainly reflects the details and texture features of the image, such as edges, contours, and local details. Therefore, the phase processing part in the spectrum dynamic aggregation module dynamically adjusts the phase information of MRI and PET features to ensure that the detail information is enhanced and optimized. Specifically, the phase processing module enhances the phase information through convolution layers, nonlinear activation functions, and phase importance weighting to ensure that details and textures are preserved during the fusion process.

[0054] Specifically, phase information is extracted from Fourier transformed MRI and PET features. and : ; ; Input phase information and The convolutional layer is used to enhance important features such as edges and details contained in the phase information. This process is expressed by the following formula: ; ; in, represents the convolutional layer used to enhance phase information, It is a non-linear activation function (such as ReLU) used to further enhance the details.

[0055] In order to dynamically adjust the contribution of phase information of different regions to details, the phase processing module introduces a phase importance weighting mechanism. This mechanism calculates the weight of phase information. , the processed phase information is weighted according to the weight to enhance the focus on the key area. The calculation formula of the phase importance weight is: ; ; The phase information is weighted using the phase importance weight: ; ; In the formula, is the phase weight generated by the convolutional layer, using The activation function controls the weights After the phase importance weighting process is completed, the obtained and is the phase information with enhanced details and textures. Finally, these weighted phase information are used to combine with the processed amplitude information to generate a new complex feature representation: ; ; The processed frequency domain features are transformed by inverse Fourier transform Return to the spatial domain and generate new spatial features. The formula for inverse Fourier transform is: ; ; After returning from the frequency domain to the spatial domain, in order to further adaptively control the strength of these features, the large kernel attention module introduces a learnable scaling parameter This scaling parameter is a learnable scalar matrix that acts on each channel and dynamically adjusts the amplitude of the output features: ; ; in, is a learnable parameter corresponding to each channel, and the optimal scaling ratio of each channel is gradually learned through training. This scaling mechanism ensures that the model can flexibly adjust the output strength under different feature conditions to prevent the feature from being too strong or too weak. At the same time, in order to retain the key information in the input features, the large core attention module finally converts the original input features into and With scaled features and Add: ; ; Thus, the final Fourier transformed feature is obtained and This operation makes the original input features and the features processed in the frequency domain and scaled and adjusted work together, preventing information loss while enhancing the representation ability of the features.

[0056] Step S5, using multi-scale convolution, channel attention and spatial attention mechanisms to process the features after Fourier transformation, to achieve adaptive fusion of features of different modalities and different regions and adaptive weight adjustment; Specifically, the characteristics after Fourier transform and First, convolution processing of different scales is performed to capture local and global information respectively. The convolution kernel extracts detailed information. The convolution kernel is used for medium-scale features, while The convolution kernel captures global context information. The specific formula is as follows:

[0057] Add the features of different scales to form the feature representation of each scale: ; In the process of calculating using the channel attention mechanism, for the features after multi-scale convolution processing, , first perform an adaptive average pooling (avg_pool) operation to extract the global information of each channel: ; ; Next, the pooled result is passed through two layers of convolution to calculate the channel attention weight and , and use the Sigmoid activation function: ; Finally, the frequency domain features are weighted: ; ; Subsequently, the spatial attention mechanism is used to spatially weight the weighted frequency domain features. First, the spatial attention weights are extracted using a convolution operation. and : ; ; Use spatial attention weights to weight frequency domain features: ; ; After channel and spatial weighting, a Convolution processes the fused frequency domain features to ensure the consistency of channel dimensions: ; Finally, the residual connection is used to transform the original frequency domain and Add it to the weighted fusion feature to generate the final frequency domain fusion feature: ; Through residual connection, the original frequency domain information is retained and the robustness of the processed fusion features is enhanced.

[0058] Step S6, using a densely connected decoder to process the fused features to obtain the final fused image representation.

[0059] In this embodiment, the encoder adopts a dense connection structure, and the dense connection structure of the decoder focuses on optimizing the quality of the fusion result. It is used to expand the fusion features to a high-dimensional space first, and then optimize the features through multiple dense connection blocks. Each dense connection block uses continuous convolution operations and feature splicing; and finally outputs the optimized fused image.

[0060] Fusion of encoder outputs First pass The convolutional layers are expanded to fit the processing dimensions of the decoder: ; In the densely connected decoding block, features are concatenated layer by layer to ensure smooth and effective information transmission during the decoding process. The layer features are , then the output of each layer can be expressed as: ; Through dense connections, each layer concatenates all previous features with the current output to form a feature representation that is accumulated layer by layer.

[0061] Final output of the densely connected decoding block Will pass a The convolution layer compresses the feature dimension to the original number of image channels and outputs the final image representation of the decoder. The fusion result is as follows: Figure 4c As shown, Figure 4a For MRI images, Figure 4b For PET images.

[0062] ; The densely connected decoder further optimizes and refines the fusion features through multiple layers of dense connections, which can effectively enhance image details and improve the visual quality of the fusion results. Specifically, the decoder first expands the low-dimensional fusion features to a higher-dimensional space, and then gradually optimizes the feature representation through multiple densely connected blocks, and finally outputs a fused image with better quality. It not only ensures the effective preservation of detail information during the fusion process, but also improves the overall quality of the fused image through progressive optimization.

[0063] Furthermore, in this embodiment, a phase and amplitude loss function is introduced, which respectively calculates the difference between the phase information and amplitude information of the fused image and the MRI and PET images, and measures the difference in phase and amplitude by the mean square error (MSE).

[0064] (a) Phase loss

[0065] Phase loss measures the difference in phase by calculating the mean square error of the phase between the fused image and the MRI and PET images in the frequency domain. The formula for phase loss is as follows: ; in represents the phase information of the fused image, and Represent the phase information of MRI and PET images respectively.

[0066] (b) Amplitude loss

[0067] Similarly, the amplitude loss measures the difference in amplitude by calculating the mean square error of the amplitudes of the fused image and the MRI and PET images in the frequency domain. The formula for the amplitude loss is as follows: ; in: represents the amplitude information of the fused image, and Represent the amplitude information of MRI and PET images respectively.

[0068] Also includes the strength loss function , gradient loss function And the complete loss function. Among them, the intensity loss function is used to measure the pixel difference between the fused image and the MRI and PET images in the spatial domain. The loss function calculates the mean square error between the fused image and the MRI image, and the mean square error between the fused image and the PET image, and gives a weight of 0.5 to the loss of the PET image.

[0069] The formula is as follows: ; in, represents the pixel value of the fused image, and Represent the pixel values ​​of MRI and PET images respectively.

[0070] The gradient loss function is used to measure the difference in gradient information between the fused image and the MRI and PET images. This helps preserve the edge and detail information in the image during the fusion process.

[0071] set up Representing the gradient operator of the image (such as the Sobel operator), the formula of the gradient loss function is as follows: ; in: represents the gradient information of the fused image, and Represent the gradient information of MRI and PET images respectively.

[0072] Finally, the total loss function can be optimized by combining phase and amplitude loss, intensity loss and sensitivity loss as follows: ; in, and is the weight parameter of each loss item, which is used to control the influence of different loss items in the total loss. The above weight parameters are set to 1, 1, 1, and 5 in sequence. Through the combination of phase and amplitude loss, intensity loss, and gradient loss, the model can simultaneously focus on the phase information, amplitude information, pixel intensity, and gradient information of the image during the fusion process, thereby more comprehensively retaining the key information of MRI and PET images in the spatial and frequency domains.

[0073] Embodiment 2 This embodiment discloses a medical image fusion system based on a large-core attention mechanism; like Figure 5 As shown, a medical image fusion system based on a large kernel attention mechanism includes: The multimodal medical image acquisition module is configured to: acquire the multimodal medical image and preprocess the acquired multimodal medical image; The large core attention mechanism processing module is configured to: input the preprocessed multimodal medical image into the dual-branch encoder, perform preliminary feature extraction through the multi-scale convolution layer; use the large core attention mechanism to perform group normalization, projection and segmentation processing on the preliminary extracted features to obtain multiple features after segmentation; The multi-scale large-kernel convolution and spatial feature extraction module is configured to: input the segmented multiple features into large-kernel attention sub-modules of different scales and corresponding spatial feature extraction convolution layers respectively to obtain global and local features of the image; and concatenate the global and local features and then project them to the original dimension; The spectrum dynamic aggregation module is configured to: perform Fourier transform on the multi-scale features projected to the original dimension using the spectrum dynamic aggregation module, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; The feature fusion and adaptive weight adjustment module is configured to: process the features after Fourier transformation by using multi-scale convolution, channel attention and spatial attention mechanisms, and realize adaptive fusion of features of different modalities and different regions and adaptive weight adjustment; The medical image representation module is configured to: process the fused features using a densely connected decoder to obtain a final fused medical image representation.

[0074] Embodiment 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0075] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a medical image fusion method based on a large-core attention mechanism as described in Example 1 of the present disclosure.

[0076] Embodiment 4 The purpose of this embodiment is to provide an electronic device.

[0077] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a medical image fusion method based on a large core attention mechanism as described in Example 1 of the present disclosure are implemented.

[0078] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0079] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0080] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A medical image fusion method based on large kernel attention mechanism, characterized in that: include: Acquire multimodal medical images, and preprocess the acquired multimodal medical images; The preprocessed multimodal medical image is input into the dual-branch encoder, and preliminary feature extraction is performed through the multi-scale convolutional layer; the preliminary extracted features are group normalized, projected and segmented using the large-core attention mechanism to obtain multiple features after segmentation; The segmented features are respectively input into large-core attention submodules of different scales and corresponding spatial feature extraction convolution layers to obtain global and local features of the image: the global and local features are concatenated and projected to the original dimension; Utilize spectrum dynamic aggregation to perform Fourier transform on the multi-scale features projected to the original dimension, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; Multi-scale convolution, channel attention and spatial attention mechanisms are used to process the features after Fourier transformation to achieve adaptive fusion of features of different modalities and regions and adaptive weight adjustment; The fused features are processed using a densely connected decoder to obtain the final fused image representation.

2. The medical image fusion method based on large-core attention mechanism according to claim 1, characterized in that: The multimodal medical images include MRI magnetic resonance imaging and PET positron emission tomography images.

3. The medical image fusion method based on large kernel attention mechanism as claimed in claim 1, characterized in that: The process of preprocessing the acquired multimodal medical image is: adjusting the input image to a standard size of 128×128, and ensuring that the pixel value range of the multimodal medical image is consistent through a normalization operation.

4. The medical image fusion method based on large kernel attention mechanism according to claim 1, characterized in that: The process of inputting the preprocessed multimodal medical image into the dual-branch encoder and performing preliminary feature extraction through the multi-scale convolutional layer is as follows: The preprocessed MRI and PET images are respectively input into their respective branch encoders, and feature extraction is performed through a multi-scale convolution layer; the multi-scale convolution uses convolution kernels of different sizes to capture feature information of different scales in the medical image; ； ； in, and Represents the multi-scale convolutional extractor for the MRI and PET branches; the output and It is the preliminary feature representation of different modalities; and They are the preprocessed MRI and PET images respectively.

5. The medical image fusion method based on large kernel attention mechanism as claimed in claim 1, characterized in that: The segmented multiple medical image features are respectively input into the large core attention submodules of different scales and the corresponding spatial feature extraction convolutional layers to obtain the global and local features of the image: In the formula, , , It is the global feature map obtained after being processed by large-core attention submodules of different scales; , , It is the local feature map obtained after processing by the convolutional layer of spatial feature extraction at different scales; LKA3, LKA5, and LKA7 are large-core attention submodules of different scales; X3, X5, and X7 are corresponding 3x3, 5x5, and 7x7 spatial feature extraction convolutional layers; , , are multiple medical image features after segmentation; The process of concatenating the global and local features and projecting them to the original dimension is: multiplying the large kernel convolution result and the spatial feature extraction result to enhance the representation capability of multi-scale features: In the formula , It is the feature representation after multi-scale feature fusion, which contains the combined information of global and local features; The concatenated multi-scale features are projected back to the original dimension through the projection convolution layer to ensure that the feature size is consistent with the input: ； ； In the formula, , It is the final feature representation obtained after the last projection layer, and its dimension is consistent with the input feature; It is the projection convolution layer.

6. The medical image fusion method based on large kernel attention mechanism according to claim 1, characterized in that: The process of using spectrum dynamic aggregation to perform Fourier transform on the multi-scale features projected to the original dimension and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain is as follows: The spectral dynamic aggregation module performs Fourier transform on the multi-scale features projected to the original dimension, transforming them from the spatial domain Convert to frequency domain , respectively obtain the amplitude and phase information of the feature; ； ； in, represents the Fourier transform operation, and Respectively represent the representation of MRI and PET image features in the frequency domain; after Fourier transform, the features are decomposed into amplitude information and phase information ; ； ； in, and Represent the amplitude information of MRI and PET images respectively, and is the phase information, is the Euler representation on the complex plane, i is the imaginary unit, is the phase angle.

7. The medical image fusion method based on large kernel attention mechanism according to claim 1, characterized in that: A medical image fusion method based on large kernel attention mechanism, also including loss function design.

8. A medical image fusion system based on large kernel attention mechanism, characterized in that: include: The multimodal medical image acquisition module is configured to: acquire the multimodal medical image and preprocess the acquired multimodal medical image; The large core attention mechanism processing module is configured to: input the preprocessed multimodal medical image into the dual-branch encoder, perform preliminary feature extraction through the multi-scale convolution layer; use the large core attention mechanism to perform group normalization, projection and segmentation processing on the preliminary extracted features to obtain multiple features after segmentation; The multi-scale large-kernel convolution and spatial feature extraction module is configured to: input the segmented multiple features into large-kernel attention sub-modules of different scales and corresponding spatial feature extraction convolution layers respectively to obtain global and local features of the image; and concatenate the global and local features and then project them to the original dimension; The spectrum dynamic aggregation module is configured to: perform Fourier transform on the multi-scale features projected to the original dimension using the spectrum dynamic aggregation module, and process the amplitude and phase information of the multi-scale features to achieve dynamic aggregation in the frequency domain; The feature fusion and adaptive weight adjustment module is configured to: process the features after Fourier transformation by using multi-scale convolution, channel attention and spatial attention mechanisms, and realize adaptive fusion of features of different modalities and different regions and adaptive weight adjustment; The medical image representation module is configured to: process the fused features using a densely connected decoder to obtain a final fused medical image representation.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in a medical image fusion method based on a large-core attention mechanism as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the medical image fusion method based on the large core attention mechanism as described in any one of claims 1-7 are implemented.

Images