Three-dimensional medical image fusion method based on deep learning

Abstract

The invention relates to a three-dimensional medical image fusion method and device based on deep learning, equipment and a medium. The method comprises the following steps: acquiring two-dimensional medical source images corresponding to at least two modes, and performing intra-mode preprocessing on the two-dimensional medical source images to obtain a single-mode two-dimensional medical original image group corresponding to each mode; performing inter-modal preprocessing on the single-modal two-dimensional medical original image group to obtain a multi-modal two-dimensional medical original image group; inputting the multi-modal two-dimensional medical original image group into a deep learning network model to generate a two-dimensional medical fusion image group; and inputting the two-dimensional medical fusion image group into an improved three-dimensional generative adversarial network taking the two-dimensional medical original image group as a real image to generate a three-dimensional medical fusion image. By adopting the method, the information content displayed by the medical fusion picture can be enriched, the interpretability of the fusion process of the medical fusion picture is improved, and the efficiency and accuracy of medical diagnosis are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image data processing, and in particular relates to a three-dimensional medical image fusion method based on deep learning. Background Art

[0002] Multimodal medical fusion images are important auxiliary tools for doctors to diagnose and treat various diseases, and play an indispensable role in auxiliary diagnosis, guiding treatment and progression detection. In the past few decades, a large number of medical image fusion methods have been proposed in the prior art. Traditional fusion methods usually use a specific transformation algorithm to decompose the image, then use a designed fusion strategy to fuse it, and finally use the inverse transformation to obtain the final fused image.

[0003] However, in the existing multimodal medical image fusion technology, the generated fusion image is basically displayed in two-dimensional images. Since the human body is three-dimensional, it is difficult for two-dimensional images to accurately reflect the three-dimensional position coordinates of the patient's lesions. In addition, the two-dimensional fusion images generated by the existing multimodal medical image fusion technology themselves display less information content and are less interpretable, which can easily lead to diagnostic errors. Summary of the invention

[0004] Based on this, it is necessary to provide a three-dimensional medical image fusion method, device, computer equipment and storage medium based on deep learning that can speed up the medical diagnosis process and improve medical decision-making capabilities in response to the above technical problems.

[0005] This application provides a three-dimensional medical image fusion method based on deep learning, including:

[0006] Acquire two-dimensional medical source images corresponding to at least two modalities, and perform intra-modality preprocessing on the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality, wherein the intra-modality preprocessing is used to improve the image quality of the two-dimensional medical source images corresponding to a single modality;

[0007] Performing inter-modality preprocessing on the single-modality two-dimensional medical original image group to obtain a multi-modality two-dimensional medical original image group, wherein the inter-modality preprocessing is used to align the two-dimensional medical source images corresponding to different modalities;

[0008] Inputting the multimodal two-dimensional medical original image group into the deep learning network model to generate a two-dimensional medical fusion image group; the two-dimensional medical fusion image group is used to represent a two-dimensional medical fusion image that fuses the features of two-dimensional medical source images of different modalities corresponding to the same spatial posture in the multimodal two-dimensional medical original image group; the deep learning network model is obtained by combining a convolutional neural network and a transformer network;

[0009] The two-dimensional medical fusion image group is input into an improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

[0010] The above-mentioned deep learning-based 3D medical image fusion method can accelerate the medical diagnosis process and improve medical decision-making ability by fusing multi-modal medical images to obtain multi-modal 3D medical fusion images. Through intra-modality preprocessing and inter-modality preprocessing, the image quality can be effectively improved and the spatial matching degree of the multi-modal 2D medical original image group can be improved.

[0011] Furthermore, through inter-modality preprocessing, the single-modality medical original image groups are paired to obtain a multi-modal two-dimensional medical original image group, which provides an efficient data structure for further image fusion and three-dimensional image generation. By combining the convolutional neural network (CNN) branch and the transformer network (Transformer) branch to extract the local and global information of the image, the visual representation ability can be enhanced and the clarity of the structure and texture of the fused image can be improved.

[0012] Furthermore, by using an improved three-dimensional generative adversarial network with a two-dimensional medical fusion image group as input images and a two-dimensional medical original image group as real images to generate a two-dimensional medical fusion image group, the quality and efficiency of three-dimensional medical image fusion are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A schematic diagram of an application environment of a solution provided by an embodiment of the present application;

[0015] Figure 2 A flowchart of a three-dimensional medical image fusion method based on deep learning provided in one embodiment of the present application;

[0016] Figure 3 A schematic flow chart of a method for generating a two-dimensional medical fusion image group provided in one embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of a deep learning network model provided for one embodiment of the present application;

[0018] Figure 5 A schematic diagram of the structure of another deep learning network model provided for one embodiment of the present application;

[0019] Figure 6 A schematic diagram of the structure of an improved three-dimensional generative adversarial network provided in one embodiment of the present application;

[0020] Figure 7 A schematic flow chart of a method for generating a multimodal two-dimensional medical original image group provided by an embodiment of the present application;

[0021] Figure 8 A schematic flow chart of another deep learning-based three-dimensional medical image fusion method provided in one embodiment of the present application;

[0022] Fig. 9 A schematic structural diagram of a three-dimensional medical image fusion device based on deep learning is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The three-dimensional medical image fusion method based on deep learning provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the computing platform 101 communicates with the medical image acquisition device 102 through a communication channel. The medical image acquisition device 102 can collect data that the computing platform 101 needs to process. The computing platform 101 can be integrated in the medical image acquisition device 102 and / or the medical service platform carried by the medical image acquisition device 102, or it can be placed on the cloud or other terminal media. The computing platform 101 can generate medical fusion images for disease diagnosis and detection based on the data collected by the medical image acquisition device 102.

[0025] Illustratively, the computing platform 101 may be, but is not limited to, a high-performance computing cluster, an edge computing platform, and a hybrid computing platform. The medical image acquisition device 102 may include, but is not limited to, an MMIF device that forms a static image based on electromagnetic energy imaging technology, an X-ray computed tomography (CT) device, a single photon emission computed tomography (SPECT) device, a positron emission tomography (PET) device, and a magnetic resonance imaging (MRI) device.

[0026] In an exemplary embodiment, Figure 2 As shown in the figure, a 3D medical image fusion method based on deep learning is provided. Figure 1 The computing platform 101 in FIG. 1 is used as an example to illustrate the invention, including:

[0027] Step S201 , obtaining two-dimensional medical source images corresponding to at least two modalities, and performing intra-modality preprocessing on the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality.

[0028] Specifically, the computing platform 101 can obtain two-dimensional medical source images corresponding to at least two modalities through a medical image acquisition device. The computing platform 101 can perform intra-modality preprocessing on the obtained two-dimensional medical source images corresponding to at least two modalities to obtain a single-modality two-dimensional medical original image group corresponding to each modality. The computing platform 101 can improve the image quality of the two-dimensional medical source image corresponding to a single modality of the at least two modalities through intra-modality preprocessing.

[0029] Exemplarily, the two-dimensional medical source image may include, but is not limited to, a CT source image, a SPECT source image, a PET source image, and an MRI source image. Intra-modality preprocessing may include, but is not limited to, denoising, contrast enhancement, and distortion correction.

[0030] Step S202 , performing inter-modality preprocessing on the single-modality two-dimensional medical original image group to obtain a multi-modality two-dimensional medical original image group.

[0031] Specifically, the computing platform 101 can perform inter-modality preprocessing on the single-modality two-dimensional medical original image group obtained after intra-modality preprocessing to obtain a multi-modality two-dimensional medical original image group. The computing platform 101 can spatially align the two-dimensional medical source images corresponding to different modalities through inter-modality preprocessing.

[0032] Exemplarily, inter-modality preprocessing may include, but is not limited to, image resampling processing, image normalization processing, and image space registration processing.

[0033] Step S203: input the multimodal two-dimensional medical original image group into the deep learning network model to generate a two-dimensional medical fusion image group.

[0034] Specifically, the computing platform 101 can input the multimodal two-dimensional medical original image group into the deep learning network model to generate a two-dimensional medical fusion image, and then construct a two-dimensional medical fusion image group based on the spatial posture corresponding to the two-dimensional medical fusion image. The two-dimensional medical fusion image group is used to represent the two-dimensional medical fusion image that fuses the features of the two-dimensional medical source images of different modalities corresponding to the same spatial posture in the multimodal two-dimensional medical original image group. The deep learning network model is obtained by combining a convolutional neural network and a transformer network.

[0035] Exemplarily, the convolutional neural network and the transformer network in the deep learning network model are multi-stage structures, and the computing platform 101 can fuse the outputs of the convolutional neural network and the transformer network in the same stage as the output of the deep learning network model in this stage.

[0036] Furthermore, the computing platform 101 can determine the number of layers of the multi-layer structure of the deep learning network model based on the downsampling multiples of each stage in the convolutional neural network and the minimum resolution of the multimodal two-dimensional medical original image group.

[0037] Step S204, inputting the two-dimensional medical fusion image group into an improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

[0038] Illustratively, the computing platform 101 may generate a 3D medical fusion image in a supervised manner through an improved 3D generative adversarial network using a 2D medical fusion image group as an input image and a 2D medical original image group as a real image.

[0039] In the above-mentioned deep learning-based three-dimensional medical image fusion method, the quality of a single-modality two-dimensional medical source image can be improved through intra-modality preprocessing, thereby ensuring the quality and consistency of the multi-modality two-dimensional medical original image group input into the deep learning model. Inter-modality preprocessing can be used to accurately align images of different modalities in space, laying the foundation for subsequent multi-modality data fusion. By inputting the multi-modality two-dimensional medical original image group into a deep learning network model that combines a convolutional neural network and a transformer network, the features of two-dimensional medical original images of different modalities can be fused to improve the accuracy and robustness of medical image analysis. By inputting the two-dimensional medical fusion image group into an improved three-dimensional generative adversarial network to generate a three-dimensional medical fusion image, the visualization effect of the medical fusion image can be enhanced, providing doctors with a comprehensive and accurate three-dimensional perspective, and improving the accuracy and real-time performance of medical diagnosis.

[0040] Furthermore, the above-mentioned deep learning-based three-dimensional medical image fusion method can provide a powerful tool for cross-modal medical image analysis research and promote the development of the field of medical image analysis.

[0041] In an exemplary embodiment, Figure 3 and Figure 4 As shown, the multimodal two-dimensional medical original image group is input into the deep learning network model to generate a two-dimensional medical fusion image group, including:

[0042] Step 301: extract a local feature map of each image in a multimodal two-dimensional medical original image group based on a convolutional neural network branch in a deep learning network model.

[0043] Specifically, the convolutional neural network branch in the deep learning network model can be divided into three stages. The first stage can be a convolutional neural network initial module including a convolution layer with a convolution kernel size of 7 and a convolution step size of 2, a maximum pooling layer, a residual convolution module, and an average pooling layer. The computing platform can achieve 4 times downsampling based on the convolutional neural network initial module, wherein the maximum pooling layer and the average pooling layer achieve two times downsampling respectively. The second and third stages can be convolutional neural network standard modules including a residual convolution module and an average pooling layer, and the computing platform can achieve 2 times downsampling based on the average pooling layer in the convolutional neural network initial module.

[0044] Indicatively, when the minimum resolution of the multimodal two-dimensional medical original image group exceeds 640×640, a trained third convolutional neural network standard module can be added to the third stage of the convolutional neural network branch as the fourth stage.

[0045] Step 302: extracting a global feature map of each image in the multimodal two-dimensional medical original image group based on a transformer network branch in a deep learning network model.

[0046] Specifically, the number of stages of the transformer network branch in the deep learning network model can be the same as the number of stages of the convolutional neural network branch. The side length of the image block corresponding to the global feature map output by the transformer network branch at the same stage in the deep learning network model can be the same as the downsampling multiple of the convolutional neural network branch. The transformer network branch in the deep learning network model is obtained by replacing the multi-head attention module in the original transformer network with a spatially reduced multi-head attention module.

[0047] Schematically, the transformer network branch in the deep learning network model can be divided into three stages, each stage including a patch embedding module and an encoder, wherein the transformer network branch can control the size of the image block corresponding to the output global feature map through the patch embedding module, and the encoder can include a normalization layer, a spatial reduction multi-head attention module and a multi-layer perceptron.

[0048] Exemplarily, the side length of the image block corresponding to the global feature map output by the first stage of the transformer network branch can be 4; the side length of the image block corresponding to the global feature map output by the second stage can be 8; and the side length of the image block corresponding to the global feature map output by the third stage can be 16.

[0049] Step 303: feature-couple the local feature map and the global feature map to generate a coupled feature map, and input the coupled feature map into a cross-scale attention module to generate a multi-scale feature map.

[0050] Specifically, the computing platform can perform feature coupling between the local feature map output by the convolutional neural network branch at the same stage and the global feature map output by the transformer network branch at the same stage through the feature fusion module in the deep learning network model; the computing platform can add the global feature map output by the transformer network branch at the same stage and the local feature map output by the convolutional neural network branch at the same stage after matching to obtain a coupled feature map.

[0051] Furthermore, the computing platform can input the coupled feature maps of each stage into the cross-scale attention module to generate multi-scale feature maps.

[0052] Step 304 , based on the multi-scale feature maps of images corresponding to different modalities to be fused in the multi-modal two-dimensional medical original image group, a two-dimensional medical fusion image is generated, and a two-dimensional medical fusion image group is constructed.

[0053] Specifically, the computing platform can input the coupled feature map output by the first stage and the multi-scale feature map output by each cross-scale attention module into the channel splicing layer to obtain the spliced ​​feature map. Then, the spliced ​​feature maps of the images corresponding to the different modalities to be fused in the multi-modal two-dimensional medical original image group are weighted summed and input into the feature decoding network to generate a two-dimensional medical fusion image. After the two-dimensional medical fusion image is spatially registered and fine-tuned, a two-dimensional medical fusion image group is constructed.

[0054] In the above-mentioned three-dimensional medical image fusion method based on deep learning, the convolutional neural network branch can effectively extract local key information such as detailed texture and edges of the lesion site. At the same time, the transformer network branch can obtain macroscopic features such as tissue distribution and overall structural morphology reflected at the overall level of the image, thereby realizing comprehensive mining of medical image features and improving the interpretability of medical image fusion.

[0055] Furthermore, by coupling and cross-scale fusion of the extracted local feature maps and global feature maps, features of different dimensions and levels can complement and fuse each other, and richer and more comprehensive image information can be obtained, providing a multi-dimensional basis for accurately judging the condition of the lesion.

[0056] In an exemplary embodiment, please refer to Figure 5 , based on the multi-scale feature maps of images corresponding to different modalities to be fused in the multi-modal two-dimensional medical original image group, a two-dimensional medical fusion image is generated, including:

[0057] The multi-scale feature maps of images corresponding to different modalities that need to be fused in the multi-modal two-dimensional medical original image group are input into the feature map fusion module to generate a fused feature map. The feature map fusion module includes a global attention mechanism submodule and a fusion submodule.

[0058] The fused feature map is input into the feature decoding network to generate a two-dimensional medical fusion image.

[0059] A two-dimensional medical fusion image group is constructed after spatial registration of the two-dimensional medical fusion images, and the two-dimensional medical fusion image group is used to represent the spatial posture of the two-dimensional medical fusion image.

[0060] In the above-mentioned deep learning-based 3D medical image fusion method, the global attention mechanism submodule in the feature map fusion module can focus on the correlation and importance distribution between the features of different modal images from a global perspective. By weighting the multi-scale feature maps of the corresponding images of different modalities, the feature information that is more critical to the fusion is highlighted, and important features are avoided from being ignored or weakened during the fusion process, so that the fusion is more in line with the needs of medical diagnosis and the advantageous information of each modality is fully integrated.

[0061] Furthermore, through the spatial registration operation, the spatial posture of the two-dimensional medical fusion image can be accurately determined to ensure the consistency of the spatial position of each fused image, thereby improving the accuracy of the three-dimensional medical fusion image.

[0062] In an exemplary embodiment, please refer to Figure 6 , the improved 3D generative adversarial network is obtained by:

[0063] An initial improved three-dimensional generative adversarial network is constructed based on the three-dimensional VEA-GAN model combined with a discriminator using the Wasserstein distance loss function with a gradient penalty method.

[0064] Specifically, the improved three-dimensional generative adversarial network includes an encoder 601, a generator 602 and a discriminator 603. Since the quality of the three-dimensional images generated unsupervisedly by the traditional three-dimensional GAN ​​model through random latent variables input into the generator is low, the quality of the three-dimensional images generated by the improved three-dimensional generative adversarial network can be improved by combining the VEA network to construct a three-dimensional VEA-GAN model. The input data of the improved three-dimensional generative adversarial network can be a two-dimensional medical fusion image group, and the real data of the improved three-dimensional generative adversarial network can be a two-dimensional medical original image group.

[0065] The pseudo three-dimensional medical fusion image generated by the generator of the improved three-dimensional generative adversarial network is two-dimensionally split at the spatial position corresponding to the image in the single-modality medical original image group to obtain the pseudo two-dimensional medical fusion image group corresponding to each single-modality medical original image group.

[0066] Each unimodal medical original image group and the pseudo two-dimensional medical fusion image group are input into the discriminator of the initial improved three-dimensional generative adversarial network to calculate the loss of the discriminator on the real image, i.e., each unimodal medical original image group, and the loss on the false image, i.e., the pseudo two-dimensional medical fusion image group, and back-propagate the loss to update the parameters of the discriminator to obtain the improved three-dimensional generative adversarial network.

[0067] Among them, the Wasserstein distance loss function expression using the gradient penalty method is as follows:

[0068]

[0069] In the formula, X is the input image, X' is the generated image, and P g To generate the distribution law of image X', P r is the distribution law obeyed by the input image X, P is the sampling self-distribution law r and distribution law P g The straight line connecting the sampling points between, λ is the penalty factor, is the gradient penalty term.

[0070] In the above-mentioned deep learning-based three-dimensional medical image fusion method, an initial improved three-dimensional generative adversarial network is constructed based on the three-dimensional VEA-GAN model combined with a discriminator of the Wasserstein distance loss function using a gradient penalty method. This can more accurately measure the distribution difference between the generated image and the real image, and can more smoothly guide the generator to generate images that are closer to the real distribution, making the generated images higher in quality and more in line with the actual situation, and reducing the possibility of problems such as mode collapse in the generated images.

[0071] Furthermore, by performing two-dimensional splitting on the pseudo three-dimensional medical fusion image generated by the generator at the spatial position corresponding to the image in the single-modality medical original image group, a pseudo two-dimensional medical fusion image group corresponding to each single-modality medical original image group is obtained, thereby more finely exploring the association and difference between the three-dimensional fusion image and the two-dimensional original single-modality image.

[0072] In an exemplary embodiment, intra-modality preprocessing is performed on the two-dimensional medical source image to obtain a single-modality two-dimensional medical original image group corresponding to each modality, including:

[0073] The two-dimensional medical source image is filtered and optimized, and the noise and artifacts in the two-dimensional medical source image are identified and removed to obtain an optimized two-dimensional medical source image.

[0074] The optimized two-dimensional medical source images are spatially registered within the modality to generate two-dimensional medical original images, so that the projection coordinates of the two-dimensional medical original images in each modality in the same spatial position are consistent in the vertical direction, and a single-modality two-dimensional medical original image group corresponding to each modality is formed.

[0075] In an exemplary embodiment, please refer to Figure 7 , performing inter-modality preprocessing on the single-modality two-dimensional medical original image group to obtain a multi-modality two-dimensional medical original image group, including:

[0076] Step 701, pairing the two-dimensional medical original images with overlapping spatial postures in the single-modality medical original image group, performing inter-modality image registration, generating a first matching multi-modality medical original image group, and identifying spatially heterotopic medical original images.

[0077] Step 702, for the two-dimensional medical original image in the other single-modality medical original image group closest to the spatially heterotopic medical original image, a reconstructed medical original image is generated at the spatial position of the spatially heterotopic medical original image based on a spline function.

[0078] Step 703 , pairing the spatially heterotopic original medical image and the reconstructed original medical image to perform inter-modality image registration to generate a second matched multi-modality original medical image group.

[0079] Step 704 : Generate a multimodal two-dimensional medical original image group by combining the first matching multimodal medical original image group and the second matching multimodal medical original image group.

[0080] In the above-mentioned three-dimensional medical image fusion method based on deep learning, the consistency and accuracy of the spatial posture of the images to be fused in the first matching multi-modal medical original image group and the second matching multi-modal medical original image group are guaranteed through image pairing, reconstruction of medical original image generation and inter-modal image registration.

[0081] In an exemplary embodiment, the deep learning network model adopts a comprehensive loss function that combines an image content loss function and an image similarity loss function;

[0082] The expression of the comprehensive loss function is as follows:

[0083] L z =ρ1×L con +ρ2×L cos

[0084]

[0085] Where, L z is the comprehensive loss function, L con is the image content loss function, Lcos is the image similarity loss function, ρ1 is the weight coefficient of the image content loss function, ρ2 is the weight coefficient of the image similarity loss function, H is the height of the two-dimensional medical original image, W is the width of the two-dimensional medical original image, N is the total number of fused modalities, I f is a two-dimensional medical fusion image, I k is the two-dimensional medical original image corresponding to the kth modality in the multi-modal two-dimensional medical original image group, is the gradient operator, α k is the modal pixel weight coefficient, β k is the modal gradient weight coefficient, cos is the vector cosine operator, γ k is the modal similarity weight coefficient.

[0086] In an exemplary embodiment, Figure 8 As shown, the three-dimensional medical image fusion method based on deep learning includes:

[0087] Step S801, acquiring two-dimensional medical source images corresponding to at least two modalities.

[0088] Step S802 , performing filtering optimization on the two-dimensional medical source image, identifying and removing noise and artifacts in the two-dimensional medical source image, and obtaining an optimized two-dimensional medical source image.

[0089] Step S803, performing intra-modality spatial registration on the optimized two-dimensional medical source images to generate two-dimensional medical original images, so that the projection coordinates of the two-dimensional medical original images in each modality in the same spatial position in the vertical direction are consistent, and a single-modality two-dimensional medical original image group corresponding to each modality is formed.

[0090] Step S804, performing inter-modality image registration on the paired two-dimensional medical original images with overlapping spatial postures in the single-modality medical original image group to generate a first matching multi-modality medical original image group, and identifying spatially heterotopic medical original images.

[0091] Step S805, for the two-dimensional medical original image in the other single-modality medical original image group closest to the spatially heterotopic medical original image, a reconstructed medical original image is generated at the spatial position of the spatially heterotopic medical original image based on a spline function.

[0092] Step S806, pairing the spatially heterotopic original medical image and the reconstructed original medical image, and performing inter-modality image registration to generate a second matched multi-modality original medical image group.

[0093] Step S807 , combining the first matching multimodal medical original image group and the second matching multimodal medical original image group to generate a multimodal two-dimensional medical original image group.

[0094] Step S808, extracting a local feature map of each image in the multimodal two-dimensional medical original image group based on the convolutional neural network branch in the deep learning network model.

[0095] Step S809: extracting a global feature map of each image in the multimodal two-dimensional medical original image group based on the transformer network branch in the deep learning network model.

[0096] Step S810, feature coupling is performed on the local feature map and the global feature map to generate a coupled feature map, and the coupled feature map is input into a cross-scale attention module to generate a multi-scale feature map.

[0097] Step S811, inputting the multi-scale feature maps of images corresponding to different modalities to be fused in the multi-modal two-dimensional original medical image group into a feature map fusion module to generate a fused feature map.

[0098] Step S812, inputting the fused feature map into the feature decoding network to generate a two-dimensional medical fusion image.

[0099] Step S813, constructing a two-dimensional medical fusion image group after spatial registration of the two-dimensional medical fusion images.

[0100] Step S814, inputting the two-dimensional medical fusion image group into an improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

[0101] The above-mentioned deep learning-based three-dimensional medical image fusion method can realize the effective optimization, fusion and three-dimensional reconstruction of multimodal two-dimensional medical images, provide richer and more accurate image data for medical diagnosis and research, and improve the efficiency and accuracy of medical diagnosis.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a deep learning-based 3D medical image fusion device for implementing the deep learning-based 3D medical image fusion method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more deep learning-based 3D medical image fusion device embodiments provided below can refer to the limitations of the deep learning-based 3D medical image fusion method above, and will not be repeated here.

[0104] In an exemplary embodiment, Fig. 9 As shown, a three-dimensional medical image fusion device 900 based on deep learning is provided, comprising:

[0105] The original image acquisition module 901 can be used to acquire two-dimensional medical source images corresponding to at least two modalities, and perform intra-modality preprocessing on the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality. The intra-modality preprocessing is used to improve the image quality of the two-dimensional medical source images corresponding to a single modality.

[0106] The multimodal preprocessing module 902 may be used to perform inter-modal preprocessing on a single-modal two-dimensional medical original image group to obtain a multi-modal two-dimensional medical original image group. The inter-modal preprocessing is used to align two-dimensional medical source images corresponding to different modalities.

[0107] The two-dimensional image fusion module 903 can be used to input the multimodal two-dimensional medical original image group into the deep learning network model to generate a two-dimensional medical fusion image group; the two-dimensional medical fusion image group is used to represent the two-dimensional medical fusion image that fuses the features of two-dimensional medical source images of different modalities corresponding to the same spatial posture in the multimodal two-dimensional medical original image group; the deep learning network model is obtained by combining a convolutional neural network and a transformer network.

[0108] The three-dimensional image fusion module 904 can be used to input the two-dimensional medical fusion image group into the improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

[0109] In one embodiment, the two-dimensional image fusion module 903 may also be used for:

[0110] Extracting local feature maps of each image in a multimodal two-dimensional medical original image group based on a convolutional neural network branch in a deep learning network model;

[0111] A global feature map of each image in a multimodal two-dimensional medical original image group is extracted based on a transformer network branch in a deep learning network model, wherein the transformer network branch is obtained by replacing a multi-head attention module in an original transformer network with a spatially reduced multi-head attention module;

[0112] The local feature map and the global feature map are feature coupled to generate a coupled feature map, and the coupled feature map is input into the cross-scale attention module to generate a multi-scale feature map;

[0113] Based on the multi-scale feature maps of images corresponding to different modalities to be fused in the multi-modal two-dimensional medical original image group, a two-dimensional medical fusion image is generated, and a two-dimensional medical fusion image group is constructed.

[0114] In one embodiment, the two-dimensional image fusion module 903 may also be used for:

[0115] Inputting multi-scale feature maps of images corresponding to different modalities to be fused in the multi-modal two-dimensional original medical image group into a feature map fusion module to generate a fused feature map, wherein the feature map fusion module includes a global attention mechanism submodule and a fusion submodule;

[0116] Inputting the fused feature map into the feature decoding network to generate a two-dimensional medical fusion image;

[0117] A two-dimensional medical fusion image group is constructed after spatial registration of the two-dimensional medical fusion images, and the two-dimensional medical fusion image group is used to represent the spatial posture of the two-dimensional medical fusion image.

[0118] In one embodiment, the deep learning-based three-dimensional medical image fusion device 900 can be used for:

[0119] An initial improved 3D generative adversarial network is constructed based on the 3D VEA-GAN model combined with a discriminator using the Wasserstein distance loss function with a gradient penalty method.

[0120] The pseudo three-dimensional medical fusion image generated by the generator of the improved three-dimensional generative adversarial network is two-dimensionally split at the spatial position corresponding to the image in the single-modality medical original image group to obtain the pseudo two-dimensional medical fusion image group corresponding to each single-modality medical original image group;

[0121] Input each unimodal original medical image group and the pseudo two-dimensional medical fusion image group into the discriminator of the initial improved three-dimensional generative adversarial network to calculate the loss of the discriminator on the real image, i.e., each unimodal original medical image group, and the loss on the fake image, i.e., the pseudo two-dimensional medical fusion image group, and back-propagate the loss, update the parameters of the discriminator, and obtain the improved three-dimensional generative adversarial network;

[0122] Among them, the Wasserstein distance loss function expression using the gradient penalty method is as follows:

[0123]

[0124] In the formula, X is the input image, X' is the generated image, and P g To generate the distribution law of image X', P r is the distribution law obeyed by the input image X, P is the sampling self-distribution law r and distribution law P g The straight line connecting the sampling points between, λ is the penalty factor, is the gradient penalty term.

[0125] In one embodiment, the original image acquisition module 901 may also be used for:

[0126] Performing filtering optimization on the two-dimensional medical source image, identifying and removing noise and artifacts in the two-dimensional medical source image, and obtaining an optimized two-dimensional medical source image;

[0127] The optimized two-dimensional medical source images are spatially registered within the modality to generate two-dimensional medical original images, so that the projection coordinates of the two-dimensional medical original images in each modality in the same spatial position are consistent in the vertical direction, and a single-modality two-dimensional medical original image group corresponding to each modality is formed.

[0128] In one embodiment, the multimodal preprocessing module 902 may also be used to:

[0129] After pairing the two-dimensional medical original images with overlapping spatial postures in the single-modality medical original image group, perform inter-modality image registration to generate a first matching multi-modality medical original image group, and identify the spatially heterotopic medical original images;

[0130] For a two-dimensional medical original image in another single-modality medical original image group that is closest to the spatially heterotopic medical original image, a reconstructed medical original image is generated at the spatial position of the spatially heterotopic medical original image based on a spline function;

[0131] After pairing the spatial heterotopic medical original image and the reconstructed medical original image, perform inter-modality image registration to generate a second matching multi-modality medical original image group;

[0132] The first matching multimodal medical original image group and the second matching multimodal medical original image group are combined to generate a multimodal two-dimensional medical original image group.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a three-dimensional medical image fusion method based on deep learning as described above are implemented.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0135] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0136] The above-mentioned embodiments only express several implementation methods of the embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the embodiments of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A three-dimensional medical image fusion method based on deep learning, characterized in that: The method comprises: Acquire two-dimensional medical source images corresponding to at least two modalities, and perform intra-modality preprocessing on the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality, wherein the intra-modality preprocessing is used to improve the image quality of the two-dimensional medical source images corresponding to a single modality; Performing inter-modality preprocessing on the single-modality two-dimensional medical original image group to obtain a multi-modality two-dimensional medical original image group, wherein the inter-modality preprocessing is used to align the two-dimensional medical source images corresponding to different modalities; Inputting the multimodal two-dimensional medical original image group into a deep learning network model to generate a two-dimensional medical fusion image group; the two-dimensional medical fusion image group is used to represent a two-dimensional medical fusion image that fuses the features of two-dimensional medical source images of different modalities corresponding to the same spatial posture in the multimodal two-dimensional medical original image group; the deep learning network model is obtained by combining a convolutional neural network and a transformer network; The two-dimensional medical fusion image group is input into an improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

2. The method according to claim 1, characterized in that The step of inputting the multimodal two-dimensional medical original image group into a deep learning network model to generate a two-dimensional medical fusion image group includes: Extracting a local feature map of each image in the multimodal two-dimensional medical original image group based on a convolutional neural network branch in the deep learning network model; Extracting a global feature map of each image in the multimodal two-dimensional medical original image group based on a transformer network branch in the deep learning network model; the transformer network branch is obtained by replacing a multi-head attention module in the original transformer network with a space-reduced multi-head attention module; Feature coupling is performed on the local feature map and the global feature map to generate a coupled feature map, and the coupled feature map is input into a cross-scale attention module to generate a multi-scale feature map; Based on the multi-scale feature maps of the images corresponding to different modalities that need to be fused in the multi-modal two-dimensional medical original image group, the two-dimensional medical fusion image is generated, and the two-dimensional medical fusion image group is constructed.

3. The method according to claim 2, characterized in that The generating the two-dimensional medical fused image based on the multi-scale feature map of the images corresponding to different modalities to be fused in the multi-modal two-dimensional medical original image group comprises: Inputting the multi-scale feature maps of the images corresponding to different modalities to be fused in the multi-modal two-dimensional original medical image group into a feature map fusion module to generate a fused feature map, wherein the feature map fusion module includes a global attention mechanism submodule and a fusion submodule; Inputting the fused feature map into a feature decoding network to generate the two-dimensional medical fusion image; The two-dimensional medical fusion image is spatially registered to construct the two-dimensional medical fusion image group, and the two-dimensional medical fusion image group is used to represent the spatial position of the two-dimensional medical fusion image.

4. The method according to claim 1, characterized in that The improved three-dimensional generative adversarial network is obtained by the following method: An initial improved 3D generative adversarial network is constructed based on the 3D VEA-GAN model combined with a discriminator using the Wasserstein distance loss function with a gradient penalty method. Performing two-dimensional splitting of the pseudo three-dimensional medical fusion image generated by the generator of the improved three-dimensional generative adversarial network at the spatial position corresponding to the image in the single-modality medical original image group to obtain a pseudo two-dimensional medical fusion image group corresponding to each of the single-modality medical original image groups; Inputting the single-modality medical original image groups and the pseudo two-dimensional medical fusion image group into the discriminator of the initial improved three-dimensional generative adversarial network to calculate the loss of the discriminator on the real image, i.e., the single-modality medical original image group, and the loss on the fake image, i.e., the pseudo two-dimensional medical fusion image group, and back-propagating the loss, updating the parameters of the discriminator, and obtaining the improved three-dimensional generative adversarial network; The Wasserstein distance loss function using the gradient penalty method is expressed as follows: In the formula, X is the input image, X' is the generated image, and P g To generate the distribution law of image X', P r is the distribution law obeyed by the input image X, P is the sampling self-distribution law r and distribution law P g The straight line connecting the sampling points between, λ is the penalty factor, is the gradient penalty term.

5. The method according to claim 1, characterized in that The intra-modality preprocessing of the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality includes: Performing filtering optimization on the two-dimensional medical source image, identifying and removing noise and artifacts in the two-dimensional medical source image, and obtaining an optimized two-dimensional medical source image; The optimized two-dimensional medical source image is subjected to intra-modality spatial registration to generate a two-dimensional medical original image, so that the projection coordinates of the two-dimensional medical original image in each modality in the vertical direction of the same spatial position are consistent, and the single-modality two-dimensional medical original image group corresponding to each modality is formed.

6. The method according to claim 5, characterized in that The inter-modality preprocessing of the single-modality two-dimensional medical original image group to obtain a multi-modality two-dimensional medical original image group includes: Performing inter-modality image registration after pairing the two-dimensional medical original images with overlapping spatial postures in the single-modality medical original image group to generate a first matching multi-modality medical original image group, and identifying spatially heterotopic medical original images; For the two-dimensional medical original image in the other single-modality medical original image group closest to the spatially heterotopic medical original image, generating a reconstructed medical original image at the spatial position of the spatially heterotopic medical original image based on a spline function; Pairing the spatial heterotopic original medical image and the reconstructed original medical image, and performing inter-modality image registration to generate a second matching multi-modality original medical image group; The multimodal two-dimensional medical original image group is generated by combining the first matching multimodal medical original image group and the second matching multimodal medical original image group.

7. The method according to any one of claims 1 to 6, characterized in that: The deep learning network model adopts a comprehensive loss function that combines the image content loss function and the image similarity loss function; The expression of the comprehensive loss function is as follows: Where, L z is the comprehensive loss function, L con is the image content loss function, L cos is the image similarity loss function, ρ1 is the weight coefficient of the image content loss function, ρ2 is the weight coefficient of the image similarity loss function, H is the height of the two-dimensional medical original image, W is the width of the two-dimensional medical original image, N is the total number of fused modalities, I f is a two-dimensional medical fusion image, I k is the two-dimensional medical original image corresponding to the kth modality in the multi-modal two-dimensional medical original image group, is the gradient operator, α k is the modal pixel weight coefficient, β k is the modal gradient weight coefficient, cos is the vector cosine operator, γ k is the modal similarity weight coefficient.

8. A three-dimensional medical image fusion device based on deep learning, characterized in that: The device comprises: An original image acquisition module, used for acquiring two-dimensional medical source images corresponding to at least two modalities, and performing intra-modality preprocessing on the two-dimensional medical source images to obtain a single-modality two-dimensional medical original image group corresponding to each modality, wherein the intra-modality preprocessing is used to improve the image quality of the two-dimensional medical source images corresponding to a single modality; A multimodal preprocessing module, configured to perform inter-modal preprocessing on the single-modal two-dimensional medical original image group to obtain a multi-modal two-dimensional medical original image group, wherein the inter-modal preprocessing is used to align the two-dimensional medical source images corresponding to different modalities; A two-dimensional image fusion module, used for inputting the multi-modal two-dimensional medical original image group into a deep learning network model to generate a two-dimensional medical fusion image group; the two-dimensional medical fusion image group is used to represent a two-dimensional medical fusion image that fuses the features of two-dimensional medical source images of different modalities corresponding to the same spatial posture in the multi-modal two-dimensional medical original image group; the deep learning network model is obtained by combining a convolutional neural network and a transformer network; The three-dimensional image fusion module is used to input the two-dimensional medical fusion image group into an improved three-dimensional generative adversarial network with the two-dimensional medical original image group as the real image to generate a three-dimensional medical fusion image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images