Heart anatomy reconstruction method based on multi-modal medical scanning imaging

Abstract

The invention relates to the technical field of heart models, in particular to a heart anatomy reconstruction method based on multi-modal medical scanning imaging, which comprises the following steps of: S1, acquiring multi-modal image data of a heart in modes of heart computed tomography, heart magnetic resonance imaging and echocardiography; s2, the obtained image data are preprocessed, the preprocessing comprises the steps of denoising, contrast enhancement, normalization and image registration, and images of different modalities can be mutually and accurately aligned; according to the multi-modal characterization learning, complementary learning can be carried out by utilizing the difference of different types of modal information, the model can better extract feature information, the dynamic behavior of the heart in the actual physiological state is predicted through the dynamic cardiac image, dynamic and functional supplementation is brought to heart model reconstruction, and the heart model reconstruction efficiency is improved. The method plays a greater role in diagnosis, treatment planning and prognosis evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of heart models, and in particular to a heart anatomical reconstruction method based on multimodal medical scanning imaging. Background Art

[0002] The difficulty in obtaining a three-dimensional model through volumetric medical image reconstruction mainly focuses on building an accurate digital heart model of the patient, which requires distinguishing the complex geometry of the heart, large structural deformations during the cardiac cycle, individual cardiac structures and surrounding tissues, as well as differences between individuals, different imaging methods, systems and centers. Existing technical research rarely focuses on direct digital heart model reconstruction.

[0003] Although there have been cases of digital heart reconstruction before, the methods are mostly multi-stage heart reconstruction, such as the solution shown in Patent Document 1, and the completeness and accuracy of digital heart reconstruction largely depends on the complexity of the algorithm's segmentation of the patient's scanned images and the reconstruction of the heart structure. Among them, when it comes to whole heart segmentation methods, deep learning-based segmentation methods are moving toward the forefront of scientific research, such as the solution shown in Patent Document 2. However, deep learning-based methods can only perform algorithmic analysis through a huge medical imaging database, and it is difficult to distinguish untrained tissues and other irrelevant areas. For computer simulation methods of generating digital heart modeling, higher resolution is usually required to form a whole heart.

[0004] Prior art literature:

[0005] Patent document 1CN109961508A Method for reconstructing static three-dimensional cardiac model

[0006] Patent document 2CN115170584A Medical image segmentation method based on multi-scale Transformer Summary of the invention

[0007] In view of the technical problems existing in the cardiac anatomy reconstruction method in the prior art, the first aspect of the present invention proposes an end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging, comprising the following steps:

[0008] S1, obtaining multimodal image data of the heart by means of cardiac computed tomography, cardiac magnetic resonance imaging and echocardiography;

[0009] S2. preprocessing the acquired image data, wherein the preprocessing includes denoising, contrast enhancement, normalization and image registration steps, so that images of different modalities can be accurately aligned with each other;

[0010] S3, based on the multimodal image data, using computer graphics technology, reconstructing the three-dimensional heart surface and generating a three-dimensional geometric model of the heart;

[0011] S4, based on the three-dimensional heart model, generate a high-precision mesh;

[0012] S5. Use deep learning technology to build a prediction model to learn the heart beat pattern in echocardiography video data and predict the mesh shape at future moments;

[0013] S6. Visualize the predicted mesh data to intuitively display the specific beating pattern and changes of the patient's heart;

[0014] Wherein, in step 3, it includes:

[0015] S31. Reconstruction of cardiac tissue area: The cardiac area is automatically segmented through UNet, and a three-dimensional model of the heart is generated using three-dimensional reconstruction technology based on the segmentation results of the cardiac area in the image.

[0016] S32. Training using the UNet algorithm: Based on the UNet architecture, prepare a cardiac image dataset for training, including a large number of echocardiographic images and their corresponding cardiac tissue region annotations, so that the UNet model can learn the correct segmentation mode, and input the preprocessed images and cardiac tissue region annotations into the improved UNet model for training;

[0017] S33. Generate a three-dimensional model of the heart: input the preprocessed image data into the trained UNet model. The UNet model will predict each frame of the image and output the segmentation result of the heart tissue area. The segmentation result of each frame of the image will be stacked in chronological order to form a three-dimensional volume data. Then, a three-dimensional reconstruction algorithm is applied to extract the outer surface of the heart from the volume data to generate a three-dimensional model of the heart.

[0018] Preferably, in step S31, the UNet algorithm gradually reduces the spatial dimension of the image through its encoder, while increasing the depth of the feature map to capture feature information of different scales. At the end of the encoder, UNet combines shallow features with deep features through jump connections to enhance the model's perception of details; the algorithm gradually restores the spatial dimension of the image through the decoder, while reducing the depth of the feature map to reconstruct the fine structure of the image.

[0019] Preferably, in step S32, the UNet algorithm uses a specific loss function to optimize the segmentation result to ensure that the model can accurately depict the heart structure, and the loss function includes a weighted cross entropy loss function, a dice loss function or an L2 regularization function.

[0020] Preferably, in step 3, step S34 is also included, in which the prediction model generated by the model is compared with the gold standard, three-dimensional scanning data of an actual heart is prepared as the gold standard, and the three-dimensional model is quantitatively evaluated according to surface distance, volume difference, and Dice similarity coefficient parameters.

[0021] Preferably, before step S3, the UNet algorithm is improved by adding a FeatPool module after every two convolution operations of the encoder.

[0022] Preferably, in the decoder part, in addition to gradually restoring the resolution of the feature map to generate the final segmentation mask, the feature map obtained by each layer of the FeatPool module is combined with the feature map in the next layer of the decoder.

[0023] Preferably, in step S5, the following steps are included:

[0024] Step S51, acquiring video data from echocardiographic data;

[0025] Step S52, decomposing the video data into a series of two-dimensional image frames;

[0026] Step S53, performing denoising, contrast enhancement, and standardization on each frame of the image so that it can be better processed by the deep learning model;

[0027] Step S54, preparing corresponding heart mesh data for each frame of image;

[0028] Step S55: Predict the mesh through feature extraction and time series modeling to infer the mesh shape at the future moment

[0029] Preferably, in step 6, the obtained mesh data is imported into a visualization tool, wherein the visualization tool includes interactive functions for rotation, scaling and slicing operations of the heart model.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] The multimodal representation learning of the present invention can perform complementary learning by utilizing the differences between different types of modal information, which is conducive to the model being able to better extract feature information and improve the accuracy of the model. In particular, multimodal cardiac images can provide more comprehensive cardiac structure, function and hemodynamic information. The patent of the present invention is not limited to surface reconstruction and post-processing technology of voxel segmentation. The outer surface of the heart is extracted from the volume data by applying a three-dimensional reconstruction algorithm through an improved UNet model, thereby generating a three-dimensional model of the heart, and predicting the dynamic behavior of the heart under actual physiological conditions through dynamic cardiac images, bringing dynamic and functional supplements to the reconstruction of the heart model, so that the model can not only represent the anatomical structure of the heart, but also simulate its functional state, thereby playing a greater role in diagnosis, treatment planning and prognosis evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in every figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which:

[0033] Figure 1 It is a comparison of CT cases using different methods to predict the entire heart surface;

[0034] Figure 2 It is a comparison of the whole heart surface reconstructed by different methods of CT images;

[0035] Figure 3 This is a comparison of the whole heart surface reconstructed by different methods of MRT images. DETAILED DESCRIPTION

[0036] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0037] A first aspect of the present invention provides an end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging, comprising the following steps:

[0038] S1, obtaining multimodal image data of the heart through cardiac computed tomography, cardiac magnetic resonance imaging and echocardiography;

[0039] Multimodal representation learning can be achieved by utilizing the differences between different modal information for complementary learning. Cardiac computed tomography and cardiac magnetic resonance imaging techniques can extract high-resolution cardiac tissue information in a non-invasive manner.

[0040] Compared with a single CT or MRI, multimodal information acquisition is more extensive and contains richer scanning information, which is conducive to the model to better extract feature information and improve the accuracy of the model. In particular, multimodal cardiac images can provide more comprehensive cardiac structure, function and hemodynamic information. For example, MRI can provide high-resolution cardiac tissue information, CT can provide information about cardiac calcification and vascular lesions, and echocardiography can obtain dynamic video data of the heart. The combination of these three modalities can help doctors assess the condition of the heart more comprehensively. Although the acquisition cost of multimodal cardiac images is high, it can reduce the cost of subsequent treatment by improving diagnostic accuracy and treatment effectiveness. In addition, multimodal images help reduce repeated examinations, thereby reducing medical costs.

[0041] S2. preprocessing the acquired image data, including denoising, contrast enhancement, normalization and image registration steps, so that images of different modalities can be accurately aligned with each other;

[0042] Preprocessing steps include noise removal, contrast enhancement, normalization, and image registration of the acquired cardiac images. These steps are designed to improve image quality and ensure that images between different modalities can be accurately aligned, thereby improving the accuracy and consistency of subsequent reconstruction results.

[0043] The main purpose of preprocessing is to eliminate interference factors in the image, make the heart image clearer and more standardized, and provide high-quality image data for input into the pre-trained UNet model. By optimizing the input, the preprocessing step can significantly improve the accuracy of heart model reconstruction, thereby providing more reliable support for clinical diagnosis and treatment.

[0044] S3, based on the multimodal image data, using computer graphics technology, reconstructing the three-dimensional heart surface and generating a three-dimensional geometric model of the heart;

[0045] In step 3, it includes S31 reconstructing the heart tissue region, S32 training using the UNet algorithm, and S33 generating a three-dimensional model of the heart.

[0046] Furthermore, S31, cardiac tissue region reconstruction: the cardiac region is automatically segmented by UNet, and a three-dimensional model of the heart is generated using three-dimensional reconstruction technology according to the segmentation result of the cardiac region in the image.

[0047] Specifically, in step S31, the UNet algorithm gradually reduces the spatial dimension of the image through its encoder, while increasing the depth of the feature map to capture feature information of different scales. At the end of the encoder, UNet combines shallow features with deep features through jump connections to enhance the model's perception of details; the algorithm gradually restores the spatial dimension of the image through the decoder, while reducing the depth of the feature map to reconstruct the fine structure of the image.

[0048] Further, S32, using the UNet algorithm for training: based on the UNet architecture, prepare a cardiac image dataset for training, including a large number of echocardiographic images and their corresponding cardiac tissue area annotations (ie, the gold standard), so that the UNet model can learn the correct segmentation pattern, and input the preprocessed images and cardiac tissue area annotations into the improved UNet model for training.

[0049] Specifically, in step S32, the UNet algorithm uses a specific loss function to optimize the segmentation result to ensure that the model can accurately depict the heart structure. The loss function includes a weighted cross entropy loss function, a dice loss function, or an L2 regularization function.

[0050] Furthermore, S33, a three-dimensional model of the heart is generated: the preprocessed image data is input into the trained UNet model, the UNet model predicts each frame of the image, outputs the segmentation result of the heart tissue area, and the segmentation result of each frame of the image is stacked in chronological order to form a three-dimensional volume data. Then, a three-dimensional reconstruction algorithm is applied to extract the outer surface of the heart from the volume data to generate a three-dimensional model of the heart.

[0051] Specifically, in step 3, step S34 is also included, in which the prediction model generated by the model is compared with the gold standard, three-dimensional scanning data of an actual heart (such as CT or MRI scan) is prepared as the gold standard, and the three-dimensional model is quantitatively evaluated based on surface distance, volume difference, and Dice similarity coefficient parameters.

[0052] In the above-mentioned embodiment, before step S3, the UNet algorithm is improved, and a FeatPool module is added after every two convolution operations of the encoder. The function of this module is to effectively reduce the spatial dimensions (i.e., height and width) of the feature map without losing too much information, while enhancing the aggregation of local information, thereby extracting more representative features. This design helps the model capture more sophisticated and complex patterns in the image and improves the feature extraction capability. By reducing the spatial dimensions of the feature map, the FeatPool module can reduce the amount of computation for subsequent operations and improve the computational efficiency of the overall model. At the same time, due to feature aggregation and dimensionality reduction, the model shows stronger stability to changes in the input image (such as noise, deformation, etc.), thereby improving the generalization ability of the model.

[0053] Preferably, in the decoder part, in addition to gradually restoring the resolution of the feature map to generate the final segmentation mask, the feature map obtained by each layer of the FeatPool module is combined with the feature map in the next layer of the decoder. This cross-layer feature fusion strategy enables the model to comprehensively utilize the information of the image at different levels and scales, which helps the model to understand the image content more comprehensively and improve the accuracy and precision of the segmentation. Through cross-layer feature fusion, the model can better capture the multi-level and multi-scale information in the image, and enhance the perception of complex structures and details. This improvement helps the model maintain a high segmentation performance when processing images with complex structures and background noise.

[0054] Advantages of the UNet algorithm: Improved model performance: Through the above improvements to the encoder and decoder, the entire UNet model has been significantly improved in terms of feature extraction, computational efficiency, generalization, and comprehensive understanding of multi-level and multi-scale information. Adaptation to complex scenarios: The improved UNet model is more suitable for processing images with complex structures and backgrounds, such as heart segmentation in medical images. It can maintain high-precision segmentation while showing stronger stability to interference factors such as noise and deformation in the image.

[0055] like Figures 1 to 3 As shown, the present invention can produce a smoother reconstructed model than the gold standard and has a smaller gap with the actual MR images with relatively large voxels.

[0056] S4, based on the three-dimensional heart model, generate a high-precision mesh;

[0057] Specifically, the output of the UNet model extracts features of the heart area, including edge, texture, shape and other information. The point coordinates on the boundary are extracted from the results of contour tracking. These point coordinates are key information for defining the surface of the heart model. Secondly, the extracted point coordinates are used for three-dimensional reconstruction to generate a three-dimensional surface model from the two-dimensional boundary information. The three-dimensional surface model is converted into mesh data, which consists of vertices (point coordinates), edges and faces. Finally, mesh data is formed through image processing technology and three-dimensional modeling algorithms.

[0058] S5. Use deep learning technology to build a prediction model to learn the heart beat pattern in echocardiography video data and predict the mesh shape at future moments;

[0059] Furthermore, in step S5, the following steps are included:

[0060] Step S51, acquiring video data from echocardiographic data;

[0061] Step S52, decomposing the video data into a series of two-dimensional image frames;

[0062] Step S53, performing denoising, contrast enhancement, and standardization on each frame of the image so that it can be better processed by the deep learning model;

[0063] Step S54, preparing corresponding heart mesh data for each frame of image;

[0064] Step S55: predict the mesh by feature extraction and time series modeling to infer the mesh shape at the future moment.

[0065] S6. Visualize the predicted mesh data to intuitively display the specific beating pattern and changes of the patient's heart;

[0066] Preferably, in step 6, the obtained mesh data is imported into a visualization tool, and the visualization tool includes interactive functions for rotation, scaling and slicing operations of the heart model.

[0067] In this way, the predicted mesh data is visualized to intuitively display the specific beating patterns and changes of the patient's heart. This helps doctors better understand the patient's heart condition and make accurate diagnosis and treatment decisions.

[0068] It should be understood that the above-mentioned mesh grid can be imported into the web page to establish digital medical services on the user side and the hospital side, and cooperate with the computer capabilities of platform equipment and big data to provide patients with a comprehensive auxiliary diagnosis system, which mainly consists of four parts: data collection equipment, core, analysis and services.

[0069] In the implemented platform, data collection devices can be selected as wearable devices and mobile applications, which can infer human vital signs data such as systolic blood pressure, diastolic blood pressure, waist circumference and glycated hemoglobin after several months to assist in inferring the beats predicted by the heart reconstruction model.

[0070] It should be understood that after completing cardiac reconstruction through clinical imaging, only the patient's current heart state can be obtained, and the specific movement patterns of the reconstructed heart cannot be predicted. However, by continuously collecting data on the user's heart beating patterns through wearable devices, the specific beating patterns and changes of the patient's heart can be viewed, which has a positive role in promoting the specific application of cardiac reconstruction models in clinical practice.

[0071] Therefore, by collecting and analyzing the patient's physiological data (such as systolic blood pressure, diastolic blood pressure, waist circumference and glycated hemoglobin), based on multimodal medical scanning imaging and real-time monitoring, combined with the computer capabilities of platform equipment and big data, the digital reconstruction model of the heart is inferred and improved, so as to more accurately simulate the function and state of the heart. It not only helps to realize the reconstruction of the heart model, but also provides important support for precision medicine, ensuring that doctors can rely on more real and dynamic heart health data when diagnosing and treating.

[0072] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. An end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging, characterized in that: The following steps are involved: S1, obtaining multimodal image data of the heart through cardiac computed tomography, cardiac magnetic resonance imaging and echocardiography; S2. preprocessing the acquired image data, wherein the preprocessing includes denoising, contrast enhancement, normalization and image registration steps, so that images of different modalities can be accurately aligned with each other; S3, based on the multimodal image data, using computer graphics technology, reconstructing the three-dimensional heart surface and generating a three-dimensional geometric model of the heart; S4, based on the three-dimensional heart model, generate a high-precision mesh; S5. Use deep learning technology to build a prediction model to learn the heart beat pattern in echocardiography video data and predict the mesh shape at future moments; S6. Visualize the predicted mesh data to intuitively display the specific beating pattern and changes of the patient's heart; Wherein, in step 3, it includes: S31. Reconstruction of cardiac tissue area: The cardiac area is automatically segmented through UNet, and a three-dimensional model of the heart is generated using three-dimensional reconstruction technology based on the segmentation results of the cardiac area in the image. S32. Training using the UNet algorithm: Based on the UNet architecture, prepare a cardiac image dataset for training, including a large number of echocardiographic images and their corresponding cardiac tissue region annotations, so that the UNet model can learn the correct segmentation mode, and input the preprocessed images and cardiac tissue region annotations into the improved UNet model for training; S33. Generate a three-dimensional model of the heart: input the preprocessed image data into the trained UNet model. The UNet model will predict each frame of the image and output the segmentation result of the heart tissue area. The segmentation result of each frame of the image will be stacked in chronological order to form a three-dimensional volume data. Then, a three-dimensional reconstruction algorithm is applied to extract the outer surface of the heart from the volume data to generate a three-dimensional model of the heart.

2. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 1, characterized in that: In step S31, the UNet algorithm gradually reduces the spatial dimension of the image through its encoder, while increasing the depth of the feature map to capture feature information of different scales. At the end of the encoder, UNet combines shallow features with deep features through jump connections to enhance the model's perception of details; the algorithm gradually restores the spatial dimension of the image through the decoder, while reducing the depth of the feature map to reconstruct the fine structure of the image.

3. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 1, characterized in that: In step S32, the UNet algorithm uses a specific loss function to optimize the segmentation result to ensure that the model can accurately depict the heart structure. The loss function includes a weighted cross entropy loss function, a dice loss function, or an L2 regularization function.

4. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 1, characterized in that: In step 3, step S34 is also included, preparing three-dimensional scanning data of an actual heart as a gold standard, comparing the prediction model generated by the model with the gold standard, and quantitatively evaluating the three-dimensional model based on surface distance, volume difference, and Dice similarity coefficient parameters.

5. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to any one of claims 1 to 4, characterized in that: Before step S3, the UNet algorithm is improved by adding a FeatPool module after every two convolution operations of the encoder.

6. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 5, characterized in that: In the decoder part, in addition to gradually restoring the resolution of the feature map to generate the final segmentation mask, the feature map obtained by the FeatPool module in each layer is combined with the feature map in the next layer of decoder.

7. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 1, characterized in that: In step S5, the following steps are included: Step S51, acquiring video data from echocardiographic data; Step S52, decomposing the video data into a series of two-dimensional image frames; Step S53: performing denoising, contrast enhancement, and standardization on each frame of the image so that it can be better processed by the deep learning model; Step S54, preparing corresponding heart mesh data for each frame of image; Step S55: predict the mesh by feature extraction and time series modeling to infer the mesh shape at the future moment.

8. The end-to-end cardiac anatomy reconstruction method based on multimodal medical scanning imaging according to claim 1, characterized in that: In step 6, the obtained mesh data is imported into a visualization tool, which includes interactive functions for rotating, scaling, and slicing the heart model.

Images