Heart multi-mode image feature extraction method, device, equipment, medium and product

Abstract

The invention discloses a heart multi-modal image feature extraction method and device, equipment, a medium and a product, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining multi-modal heart image data, patient information and case information in a multi-center imaging department and an ultrasound department; labeling the multi-modal heart image data, and constructing a multi-modal iconography feature database; establishing a database platform comprising a server storage control terminal and a data screening and uploading terminal; opening a communication link after verifying the identity of the requester; training is carried out based on the marked multi-mode heart data and a deep learning algorithm so as to obtain heart structure features of heart ultrasound and nuclear magnetic images through coronary angiography (CCTA) image data; finally, the system is deployed and applied clinically. According to the method, the heart structure features of the other two modal images are predicted based on the single-modal CCTA image data, the diagnosis range based on the CCTA image data is expanded, the medical overhead is saved, and the diagnosis and treatment efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a cardiac multi-modal image feature extraction method, device, equipment, medium and product. BACKGROUND

[0002] The structure and function of the heart have important clinical diagnostic significance under various clinical conditions. At present, traditional echocardiography is widely used for non-invasive assessment of cardiac functional parameters such as myocardial strain, left ventricular ejection fraction, cardiac output, and chamber volume. However, due to the complex structure and geometry of the heart, single modality imaging data has its advantages and limitations in assessing cardiac structure, function, and perfusion. In addition, in patients with coronary heart disease, coronary computed tomography angiography (CCTA) is a commonly collected imaging data. However, coronary heart disease is not only related to the degree of coronary artery stenosis, but also related to many factors such as valve status, myocardial strain, myocarditis, and cardiac ejection function.

[0003] CCTA is usually collected only at specific phases of diastole, so it cannot assess myocardial strain and other information available from traditional ultrasound and cardiac magnetic resonance (CMR) methods. At the same time, cardiac magnetic resonance can accurately assess chamber function and obtain diagnostic information such as myocarditis, myocardial ischemia, and myocardial strain that is closely related to coronary heart disease without geometric assumptions, but due to its high time consumption and inapplicability to patients with metal heart stents, it cannot be widely collected in patients with coronary heart disease. Each imaging method has complementary in the diagnosis and management of cardiac pathology, and how to synchronously obtain cardiac feature information of other imaging modalities through conventional single modality CCTA data is of great significance for accurate diagnosis of cardiac diseases. SUMMARY

[0004] The purpose of the present application is to provide a cardiac multi-modal image feature extraction method, device, equipment, medium and product, which can obtain features of other modalities through CCTA image data only.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a cardiac multi-modal image feature extraction method, comprising the following steps:

[0007] The multi-center imaging department and the ultrasonic department acquire multi-modal cardiac imaging data and patient information and case information; the multi-modal cardiac imaging data includes cardiac ultrasound imaging, cardiac nuclear magnetic imaging and CCTA imaging data.

[0008] The multi-modal cardiac imaging data is labeled and a multi-modal imaging feature database is constructed; the labels labeled on the multi-modal cardiac imaging data are cardiac contours, intracavity contours, myocarditis lesion region contours and respective modalities that can reflect cardiac physiological and pathological structure features.

[0009] A database platform is built to perform data screening and storage management on the uploaded multi-modal cardiac imaging data; the database platform includes a data screening upload terminal and a server storage control end.

[0010] An audit request for accessing the server storage control end is received and the identity of the requester is verified; information is fed back to the requester who passes the audit and a communication link is opened; a data upload interactive interface is displayed on the data screening upload terminal of the requester to enable the requester to upload the multi-modal cardiac imaging data of the patient.

[0011] Based on the labeled multi-modal cardiac data and deep learning algorithms, cardiac structure-function features are extracted and mapped, image modality conversion and registration models are trained and model parameters are optimized.

[0012] The image modality conversion and registration model is used to convert and deform the input CCTA imaging data to obtain corresponding cardiac ultrasound imaging and cardiac nuclear magnetic imaging cardiac structure features.

[0013] Optionally, the multi-modal cardiac imaging data is labeled and a multi-modal imaging feature database is constructed, specifically including the following steps:

[0014] The multi-modal cardiac imaging data is subjected to quality screening and data cleaning; the data cleaning includes denoising, correction, uniform size and gray scale range.

[0015] The multi-modal cardiac imaging data is labeled according to the standard cardiac region of interest; the labeling types include structure labeling and function labeling, and the labeling content is determined according to the patient's condition and the modality characteristics.

[0016] The labeled data is audited and data augmentation is performed to construct a multi-modal imaging feature database.

[0017] Optionally, if the patient has no myocarditis related disease, the cardiac contours and intracavity contours are labeled in the three modal images of cardiac ultrasound imaging, cardiac nuclear magnetic imaging and CCTA imaging data; if the patient has myocarditis disease, the myocarditis region range, myocardial tissue lesion type and valve structure abnormality are additionally labeled.

[0018] Optionally, the uploaded data is encrypted by using a symmetric encryption algorithm or an asymmetric encryption algorithm, the encrypted data is transmitted to the server storage control end through the network communication module, and when the data is received at the server storage control end, the encrypted data is decrypted, and a data encryption transmission log is generated and saved to ensure the security of all patient data and personal information.

[0019] Optionally, the corresponding cardiac structure-function is feature extracted and mapped, including feature extraction and mapping training of paired data of CCTA images and cardiac ultrasound images, and CCTA images and cardiac nuclear magnetic images, respectively.

[0020] The paired data of CCTA images and cardiac nuclear magnetic images are feature extracted and mapped, specifically including the following steps:

[0021] The CCTA images are converted into cardiac nuclear magnetic modality images by the image modality conversion network, and the multi-modality registration is converted into a single-modality registration problem.

[0022] The pixel difference between the cardiac nuclear magnetic modality image obtained after modality conversion and the original cardiac nuclear magnetic image is measured by using the L1 norm, and the registration deformation field is constrained by combining the smooth regularization loss.

[0023] The mutual information of the input and output image blocks is maximized by the image block contrast loss, and the anatomical structure consistency is maintained.

[0024] The high-precision spatial alignment of CCTA images and cardiac nuclear magnetic images is realized by end-to-end collaborative optimization.

[0025] The paired data of CCTA images and cardiac ultrasound images are feature extracted and mapped, specifically including the following steps:

[0026] The cardiac three-dimensional model is established based on the CCTA images, which is used as a standard template for heart ultrasound image registration.

[0027] High-entropy frame images are selected from the cardiac ultrasound image sequence, and the information entropy principle is used to ensure that the images contain clear anatomical features.

[0028] The mapping relationship between the cardiac three-dimensional model simulation signal and the real cardiac ultrasound signal is established by a linear combination model or a polynomial nonlinear combination model, and a similarity measure LC is defined 2 .

[0029] The global search strategy is used to locate the registration area, and the local optimization is combined to realize the registration of dynamic cardiac ultrasound images and static CCTA images.

[0030] Optionally, the contrast loss function is as follows:

[0031]

[0032] wherein, L PatchNCE is a contrast loss function value, E x is an expectation item, L is a number of levels for image segmentation, is an output image block feature, S l is a number of image blocks in the lth level, is a corresponding input image block feature, is other input image block features, is an InfoNCE loss.

[0033] In a second aspect, the application provides a cardiac multi-modal image feature extraction device, comprising:

[0034] A multi-modal data acquisition module is configured to acquire multi-modal cardiac image data and patient information and case information in multi-center imaging departments and ultrasound departments; the multi-modal cardiac imaging data comprises cardiac ultrasound images, cardiac magnetic resonance images, and CCTA image data.

[0035] A multi-modal data preprocessing module is configured to label the multi-modal cardiac image data and construct a multi-modal imaging feature database; the labels labeled on the multi-modal cardiac image data are cardiac contours, myocarditis lesion region contours, chamber inner contours, and cardiac physiological and pathological structure features that can be reflected under respective modalities.

[0036] A database platform construction module is configured to build a database platform, perform data screening on uploaded multi-modal cardiac image data, and perform storage management; the database platform comprises a data screening upload terminal and a server storage control end.

[0037] A communication link establishment module is configured to receive an audit request for accessing the server storage control end and verify the identity of the requester, feed back information to the requester who passes the audit and open a communication link, and display a data upload interactive interface on the data screening upload terminal of the requester to enable the requester to upload multi-modal cardiac image data of a patient.

[0038] A model training and optimization module is configured to perform feature extraction and mapping on corresponding cardiac structure-function based on labeled multi-modal cardiac data and a deep learning algorithm, train an image modality conversion and registration model, and optimize model parameters.

[0039] An image modality conversion and registration module is configured to perform modality conversion and deformation on input CCTA image data by using the image modality conversion and registration model to obtain cardiac structure features of corresponding cardiac ultrasound images and cardiac magnetic resonance images.

[0040] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the cardiac multi-modal image feature extraction method described above.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the cardiac multi-modal image feature extraction method described above.

[0042] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of the cardiac multi-modal image feature extraction method described above.

[0043] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0044] The present application provides a cardiac multi-modal image feature extraction method, device, equipment, medium and product, which comprises the following steps: first, acquiring multi-modal cardiac image data and patient information and case information in multi-center imaging departments and ultrasonic departments; then, labeling the multi-modal cardiac image data and constructing a multi-modal imaging feature database; then, the labels labeled on the multi-modal cardiac image data are cardiac contours, intracavity contours and cardiac structure features that can be reflected under respective modalities; subsequently, a database platform comprising a data screening upload terminal and a server storage control terminal is built, the uploaded multi-modal cardiac image data is subjected to data screening and storage management; an audit request for accessing the server storage control terminal is received and the identity of the requester is verified, information is fed back to the requester who passes the audit and a communication link is opened, a data upload interactive interface is displayed on the data screening upload terminal of the requester for the requester to upload the multi-modal cardiac image data of the patient; then, based on the labeled multi-modal cardiac data and a deep learning algorithm, cardiac structure-function features are extracted and mapped, an image modality conversion and registration model is trained and model parameters are optimized; finally, the input CCTA image data is subjected to modality conversion and deformation by using the image modality conversion and registration model, and cardiac structure features of corresponding cardiac ultrasound images and cardiac magnetic resonance images are obtained. The above-mentioned scheme of the present application collects rich data through a multi-center data acquisition module, solves the problem of single data source; a high-quality database is constructed through label labeling and other preprocessing, providing a reliable basis for subsequent training; the database platform built realizes efficient data management and sharing; cross-modality feature mapping is realized through a deep learning algorithm, cardiac structure features of other modalities can be obtained only by using CCTA image data, and the diagnosis range based on CCTA image data is expanded; when it is deployed and applied in the clinic, medical expenses can be saved, and the efficiency and accuracy of cardiac disease diagnosis and treatment are improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 A flow chart of a heart multi-modal image feature extraction method provided by an embodiment of the present application.

[0047] Figure 2 A flow chart of step A2 in a heart multi-modal image feature extraction method provided by an embodiment of the present application.

[0048] Figure 3 A flow chart of feature extraction and mapping training on paired data of CCTA images and cardiac magnetic resonance images in a heart multi-modal image feature extraction method provided by an embodiment of the present application.

[0049] Figure 4 A flow chart of feature extraction and mapping training on paired data of CCTA images and cardiac ultrasound images in a heart multi-modal image feature extraction method provided by an embodiment of the present application.

[0050] Figure 5 A functional module schematic diagram of a heart multi-modal image feature extraction device provided by an embodiment of the present application.

[0051] Figure 6 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0054] A heart multi-modal image feature extraction method provided by an embodiment of the present application, in an exemplary embodiment, as shown in Figure 1 includes the following steps:

[0055] A1, acquiring multi-modal cardiac imaging data and patient information and case information in a multi-center imaging department and ultrasound department; the multi-modal cardiac imaging data includes cardiac ultrasound imaging, cardiac magnetic resonance imaging and CCTA imaging data.

[0056] A2, labeling the multi-modal cardiac imaging data and constructing a multi-modal imaging feature database; the labels labeled on the multi-modal cardiac imaging data are cardiac contours, intracardiac contours, myocardial inflammation lesion region contours and respective modalities that can reflect the physiological and pathological structure features of the heart. In this embodiment, a suitable database management system is selected, a classification architecture including a patient information table, a multi-modal imaging data table and a labeling table is designed, the data of each table is associated with the patient ID information, and a cardiac multi-modal data local management system with expandability, efficient retrieval and safe storage functions is constructed.

[0057] A3, building a database platform to perform data filtering and storage management on the uploaded multi-modal cardiac imaging data; the database platform includes a data filtering upload terminal and a server storage control end. The server storage control end is used to receive data filtering, data filtering upload terminal sends data, and stores and manages the data, including establishing a storage index, allocating storage resources, monitoring a storage state, issuing an expansion warning when the storage resources are insufficient; and providing a data access interface for identity verification and permission management of users accessing data; the data filtering upload terminal is used to obtain original multi-modal cardiac imaging data, filter the original multi-modal cardiac imaging data according to a preset data filtering rule, the filtering rule includes data format verification, data integrity check and data quality evaluation; the filtered multi-modal cardiac imaging data is uploaded to the server storage control end through a communication module, and the data is encrypted during the uploading process, a data uploading log is generated and saved.

[0058] A4, receiving an audit request expected to access the server storage control end and verifying the identity of the requester, and feeding back information to the requester who passes the audit and opening a communication link. A data uploading interactive interface is displayed on the data filtering upload terminal of the requester to allow the requester to upload the patient's multi-modal cardiac imaging data. The audit request expected to access the server storage control end includes the identity information of the requester, and the identity information at least includes one or more of the requester's name, the requester's work number, the requester's unit, the requester's department and the requester's title. The data uploading interactive interface at least includes patient information, cardiac multi-modal imaging data, patient medical information and patient follow-up information.

[0059] A5, based on the labeled multi-modal cardiac data and deep learning algorithm, the corresponding cardiac structure-function is extracted and mapped, the image modality conversion and registration model is trained and the model parameters are optimized.

[0060] A6, the input CCTA image data is modality-converted and deformed by using an image modality conversion and registration model to obtain the physiological and pathological structure features of the heart corresponding to the cardiac ultrasound image and the cardiac magnetic resonance image. More abundant information is provided for clinicians to diagnose heart-related diseases, and the medical expenses and diagnosis time of patients are reduced.

[0061] In an exemplary embodiment, as shown in Figure 2 A2 specifically includes the following steps:

[0062] A21, quality screening is performed on the multi-modal cardiac image data, and data cleaning is performed; the data cleaning includes denoising, correction, uniform size and gray scale range. The collected and sorted cardiac ultrasound image, CCTA image and cardiac magnetic resonance image, and patient basic information and clinical test data are subjected to quality screening to remove erroneous data; the data is ensured to be complete and accurate and is classified and numbered; then the data is cleaned, a filtering algorithm is used to remove noise, brightness and contrast are corrected, specific artifacts are processed for different modal data, a filtering and enhancement algorithm is used for ultrasound image, artifacts are corrected and noise is reduced for the cardiac ultrasound image, magnetic field deviation is compensated for the cardiac magnetic resonance image, and the size and gray scale range of each modal data are unified.

[0063] A22, the multi-modal cardiac image data is labeled according to the standard cardiac region of interest; the labeling types include structure labeling and function labeling, and the labeling content is determined according to the patient condition and the modal characteristics. Specifically, professional medical image physicians label the cardiac region of interest according to the unified standard, and the labeling quality is ensured through cross-auditing by multiple people and expert discussion; then the image segmentation algorithm is used to separate the region of interest from the background.

[0064] A23, the labeled data is audited and data augmentation is performed to construct a multi-modal imaging feature database. The labeling quality is ensured through cross-auditing by multiple people and expert discussion, a professional labeling tool is selected, and the labeling type and standard are determined. In an exemplary embodiment, the labeling types include structure labeling, function labeling, pathology labeling and time sequence labeling.

[0065] First, a class label (such as normal / abnormal, lesion type) is assigned to the whole image or sequence, i.e. classification labeling; second, object detection is performed, the position (bounding box) and class of the lesion region are marked, each pixel is classified, and different tissues or lesions are distinguished.

[0066] The annotation standards for different modalities of data are different. First, the annotation should be based on clinical guidelines and consensus, and the same annotation software should be used for annotation. Annotation tools: such as ITK-SNAP (segmentation), 3D Slicer, LabelMe; quality control requirements, inter / intra-group annotator consistency and review by chief physician; ethical and privacy standards, etc.

[0067] The three modalities of data need to be annotated for heart contour and chamber contour. For data such as MRI and ultrasound that contain more myocardial tissue information and valve structure information, the type and area of myocardial tissue lesions, myocardial strain and chamber volume measurement, and valve structure and lesions also need to be annotated. Specifically, if the patient has no myocarditis-related disease, the heart contour and chamber contour are annotated in the three modalities of images: cardiac ultrasound images, cardiac MRI images, and CCTA image data. If the patient has myocarditis disease, the myocarditis area range, myocardial tissue lesion type, and valve structure abnormalities are additionally annotated.

[0068] Then, experienced imaging and ultrasound professionals manually annotate the three modalities of cardiac imaging data according to the annotation standards on the annotation tool. After annotation, another professional verifies and checks the accuracy of the annotation. If necessary, the annotated data is rotated, flipped, and other data augmentation is performed. Finally, the annotated data is saved in a specific format, and a local data management system is established for storage, retrieval, and backup.

[0069] It should be noted that during the acquisition of cardiac ultrasound data, the initial direction of the cardiac ultrasound probe is obtained with the help of a magnetic tracking system to provide a relatively accurate starting pose for subsequent image registration. For direction deviation caused by special anatomical structure of the heart or imaging needs, the head-tail axis angle of the cardiac ultrasound probe can be manually adjusted to ensure the accuracy of the initial estimate, provide a good start for the subsequent optimization process, and reduce the computational load and time cost of the overall registration.

[0070] In this scheme, in order to protect the safety of patient data and personal information, symmetric encryption algorithm or asymmetric encryption algorithm is used to encrypt the uploaded data during data transmission, the encrypted data is transmitted to the server storage control end through the network communication module, and the encrypted data is decrypted when the server storage control end receives the data, while generating a data encryption transmission log and saving it, ensuring that the data is not illegally obtained and misused.

[0071] The registration process is carried out in two aspects, one is the registration of CCTA-magnetic resonance images, and the other is the registration of CCTA and ultrasound images, and the final goal is to let CCTA learn the characteristics of the other two modal data; in this embodiment, the step A5 "feature extraction and mapping of corresponding cardiac structure-function" includes: feature extraction and mapping training of paired data of CCTA images and cardiac ultrasound images, and feature extraction and mapping training of paired data of CCTA images and cardiac magnetic resonance images.

[0072] In this embodiment, the feature extraction and mapping training of the paired data of the CCTA image and the cardiac magnetic resonance image is carried out, as shown in Figure 3 , including the following steps:

[0073] B1, converting the CCTA image into a cardiac magnetic resonance modality image through an image modality conversion network, and converting the multi-modal registration into a single-modal registration problem.

[0074] B2, adopting L1 norm to measure the pixel difference between the cardiac magnetic resonance modality image obtained after modality conversion and the original cardiac magnetic resonance image, and combining a smooth regularization loss to constrain the registration deformation field.

[0075] B3, maximizing the mutual information of the input and output image blocks through the image block comparison loss, and keeping the anatomical structure consistency.

[0076] B4, realizing high-precision spatial alignment of the CCTA image and the cardiac magnetic resonance image through end-to-end collaborative optimization.

[0077] The feature extraction and mapping training of the paired data of the CCTA image and the cardiac ultrasound image is carried out, as shown in Figure 4 , including the following steps:

[0078] C1, establishing a cardiac three-dimensional model based on the CCTA image as a standard template for the registration of the cardiac ultrasound image.

[0079] C2, selecting a high-entropy frame image from the cardiac ultrasound image sequence, and ensuring that the image contains clear anatomical features based on the information entropy principle.

[0080] C3, establishing a mapping relationship between the cardiac three-dimensional model simulation signal and the real cardiac ultrasound signal through a linear combination model or a polynomial type nonlinear combination model, and defining a similarity measure LC 2 .

[0081] C4, adopting a global search strategy to locate the registration area, and combining local optimization to realize the registration of the dynamic cardiac ultrasound image and the static CCTA image.

[0082] In an exemplary embodiment, the image registration process of CCTA images and cardiac nuclear magnetic images includes: first converting the multi-modal image registration into a single-modal problem, converting the source modal image into the target modal through the image modal conversion network, and then using the single-modal similarity measure (such as the pixel-level loss) for registration; by maximizing the mutual information between the input image block and the output image block, the image modal conversion network is forced to keep the consistency of the anatomical structure shape, and the generation of artifacts is avoided; the L1 norm is used to measure the pixel difference between the new image after modal conversion and the target image y, promote feature matching, and ensure that the features (such as brightness, texture) of the image after modal conversion in the target modal are close to the real target image; the U-Net structure is used to predict the registration deformation field, align the source image (or the image after modal conversion) with the target image, and introduce the smooth regularization loss (L2 gradient constraint) to prevent the deformation field from being excessively distorted; based on the image block comparison learning of the source image after modal conversion and the target image, the local alignment loss of the image block level is aligned in detail, and the global alignment loss of the overall shape (such as the organ contour) consistency is constrained by the L1 distance, and the final target function integrates the contrast loss (L2 gradient constraint), the pixel loss (L1 norm), and the local and global alignment loss to realize multi-task cooperation. The contrast loss function is as follows:

[0083]

[0084] wherein L PatchNCE is the contrast loss function value, E x is the expectation item, L is the number of levels divided for the image, is the output image block feature, S l is the number of image blocks in the lth level, is the corresponding input image block feature, is other input image block feature, is the InfoNCE loss.

[0085] Through the end-to-end cooperative optimization mechanism of the image modal conversion network and the registration network, high-precision spatial alignment of cross-modal images (such as CMR and CCTA) is realized, and the registration problem of multi-modal medical images (such as CMR and CCTA) is solved at one time, which is faster and more accurate than the traditional step-by-step processing.

[0086] In another exemplary embodiment, the image registration process of CCTA images and cardiac ultrasound images includes: establishing a cardiac model based on CCTA image data, providing a standard template for the morphology of each different section at different time phases obtained by real-time scanning of the ultrasound, and subsequent automatic measurement, and providing a basis for obtaining a specific standard section; in addition, based on automatic frame selection, similarity measure (LC 2Methods) and optimization strategy to register large deformation model (cardiac ultrasound image). General CCTA image acquisition is single phase, so the segmented heart is equivalent to a non-deformable rigid model. In contrast, cardiac ultrasound and magnetic resonance images can capture all sequences of the cardiac cycle and are dynamic models that reflect the systolic / diastolic dynamic process of the heart. Therefore, they can be considered as large deformation models.

[0087] First, each frame of image in the cardiac ultrasound sequence is analyzed, and based on the principle of information entropy, a high-entropy frame is accurately selected from a large number of images, so that the image can clearly show the key anatomical features of myocardial tissue, cardiac chambers and valves, and ensure that the ultrasound image entering the registration link has clear and identifiable anatomical features.

[0088] Further, based on the linear combination model, it is assumed that the ultrasound intensity can be represented as a linear combination of the CCTA simulation signal, thereby establishing a link between the cardiac CCTA simulation signal and the real cardiac ultrasound signal, and providing a theoretical basis for similarity measurement. By matrix inversion calculation, the weight is calculated to minimize the error between the simulation signal and the real ultrasound signal, and then the similarity measurement LC2 is defined. This solving method gives LC2 strong robustness to changes in brightness and contrast of ultrasound images. Even under different imaging conditions, if the ultrasound image has brightness unevenness and contrast difference, the LC2 measurement can still accurately measure the similarity between images, ensuring the reliability of registration. In another alternative embodiment, a polynomial nonlinear combination model can be used to establish the mapping relationship between the cardiac three-dimensional model simulation signal and the real cardiac ultrasound signal.

[0089] Finally, through a global search strategy, the potential registration area is quickly located in a larger range. Through this global search strategy, the search range of subsequent local optimization can be quickly narrowed, blind search in the entire image space can be avoided, and the registration efficiency can be improved, thereby providing a guarantee for quickly obtaining accurate registration results. The global search strategy is to use the exhaustive translation space method and combine the constraint conditions of the cardiac skin surface to quickly locate the potential registration area in a larger range. Through this global search strategy, the search range of subsequent local optimization can be quickly narrowed, blind search in the entire image space can be avoided, and the registration efficiency can be improved, thereby providing a guarantee for quickly obtaining accurate registration results.

[0090] Through the above image registration process, the efficiency and accuracy of the registration of cardiac CCTA with cardiac ultrasound data and cardiac magnetic resonance data are comprehensively guaranteed. The model can make the CCTA mapping obtain the physiological and pathological structure characteristics of the heart under the other two image modalities. It is expected that patients can only do CCTA to simultaneously obtain diagnostic information about the heart from the other two image modalities, thereby providing strong technical support for the diagnosis and interventional treatment of heart disease.

[0091] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the cardiac multi-modal image feature extraction method described above. The device provides a solution to the implementation scheme as described in the above method. In one exemplary embodiment, as shown in Figure 5 FIG. 1, a cardiac multi-modal image feature extraction device is provided, comprising the following functional modules:

[0092] A multi-modal data acquisition module is configured to acquire multi-modal cardiac image data and patient information and case information in multi-center imaging departments and ultrasound departments. The multi-modal cardiac image data includes cardiac ultrasound images, cardiac magnetic resonance images, and CCTA image data.

[0093] A multi-modal data preprocessing module is configured to label the multi-modal cardiac image data and construct a multi-modal image feature database. The labels labeled on the multi-modal cardiac image data are cardiac contours, intracavity contours, myocarditis lesion region contours, and respective modalities that can reflect the physiological and pathological structure features of the heart.

[0094] A database platform construction module is configured to build a database platform, perform data screening on the uploaded multi-modal cardiac image data, and perform storage management. The database platform includes a data screening upload terminal and a server storage control end.

[0095] A communication link establishment module is configured to receive an audit request for accessing the server storage control end and verify the identity of the requester, feed back information to the requester who passes the audit and open a communication link, and display a data upload interaction interface on the data screening upload terminal of the requester for the requester to upload the multi-modal cardiac image data of the patient.

[0096] A model training and optimization module is configured to extract and map the corresponding cardiac structure-function based on the labeled multi-modal cardiac data and deep learning algorithms, train the image modality conversion and registration model, and optimize the model parameters.

[0097] An image modality conversion and registration module is configured to convert and deform the input CCTA image data using the image modality conversion and registration model to obtain the cardiac structure features of the corresponding cardiac ultrasound images and cardiac magnetic resonance images.

[0098] In the scheme, in order to guarantee the security of patient data and personal information, as an optional implementation, a data encryption transmission module is further included, which is configured to perform encryption processing on the uploaded data by using a symmetric encryption algorithm or an asymmetric encryption algorithm when data transmission is performed, transmit the encrypted data to the server storage control end through the network communication module, and perform decryption processing on the encrypted data when the server storage control end receives the data, while generating and saving a data encryption transmission log, so as to guarantee that the data cannot be illegally obtained and misused.

[0099] Of course, Figure 5 The device architecture shown is only exemplary, and when different functions are implemented, one or at least two components in the device shown can be omitted Figure 5 according to actual needs.

[0100] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown. Figure 6 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor, and can implement the heart multi-modal image feature extraction method provided in the foregoing embodiments.

[0101] Those skilled in the art can understand, Figure 6 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0103] In an exemplary embodiment, a computer readable storage medium storing a computer program is provided, the computer program, when executed by a processor, implements the steps of any of the above method embodiments.

[0104] In an exemplary embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0105] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0106] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0107] The database involved in each of the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in each of the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, and the like, without being limited thereto.

[0108] Each of the technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, each of the technical features in the above embodiments is not described in all possible combinations, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present disclosure.

[0109] The principles and implementation modes of the present application are described by applying specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will have changes. In conclusion, the content of the present specification should not be understood as a limitation of the present application.

Claims

1. A method for extracting multimodal cardiac imaging features, characterized in that, include: Multimodal cardiac imaging data, patient information, and case information are acquired in radiology and ultrasound departments at multiple centers; the multimodal cardiac imaging data includes cardiac ultrasound images, cardiac MRI images, and CCTA images. The multimodal cardiac imaging data are labeled to construct a multimodal imaging feature database; the labels on the multimodal cardiac imaging data are cardiac outline, intracavitary outline, myocarditis lesion area outline, and cardiac physiological and pathological structural features that can be reflected in each modality; A database platform is established to filter and manage the uploaded multimodal cardiac imaging data; the database platform includes a data filtering and uploading terminal and a server storage control terminal. It receives the review request for accessing the server storage control terminal and verifies the identity of the requester. It provides feedback to the approved requester and opens the communication link. It displays the data upload interaction interface on the requester's data filtering and uploading terminal for the requester to upload the patient's multimodal cardiac image data. Based on labeled multimodal cardiac data and deep learning algorithms, feature extraction and mapping of the corresponding cardiac structure-function are performed, image modality conversion and registration models are trained, and model parameters are optimized. Image modality transformation and registration models are used to perform modality transformation and deformation on the input CCTA image data to obtain the cardiac structural features of the corresponding echocardiogram and MRI images.

2. The method for extracting multimodal cardiac imaging features according to claim 1, characterized in that, The multimodal cardiac imaging data are labeled and annotated to construct a multimodal imaging feature database, specifically including: The multimodal cardiac imaging data undergoes quality screening and data cleaning; the data cleaning includes noise reduction, correction, and standardization of size and grayscale range. The regions of interest in the heart are labeled on the multimodal cardiac imaging data according to the specifications; the labeling types include structural labeling and functional labeling, and the labeling content is determined according to the patient's condition and modal characteristics; The labeled data were reviewed and augmented to construct a multimodal image feature database.

3. The method for extracting multimodal cardiac imaging features according to claim 1, characterized in that, If the patient does not have myocarditis-related disease, the cardiac outline and intracavitary outline are marked in the three modal images: echocardiography, MRI, and CCTA. If the patient has myocarditis, the extent of the myocarditis area, the type of myocardial lesion, and valvular structural abnormalities are additionally marked.

4. The method for extracting multimodal cardiac imaging features according to claim 1, characterized in that, The uploaded data is encrypted using either asymmetric or asymmetric encryption algorithms. The encrypted data is then transmitted to the server storage control terminal via a network communication module. Upon receiving the data, the server storage control terminal decrypts the encrypted data and generates and saves a data encryption transmission log to ensure the security of all patient data and personal information.

5. The method for extracting multimodal cardiac imaging features according to claim 1, characterized in that, Feature extraction and mapping of the corresponding cardiac structure-function were performed, including feature extraction and mapping training of paired data of CCTA images and echocardiogram images, and CCTA images and cardiac MRI images, respectively. Feature extraction and mapping training were performed on paired CCTA and cardiac MRI images, specifically including: The image modality conversion network converts CCTA images into cardiac MRI modal images, transforming multimodal registration into a single-modal registration problem. The L1 norm is used to measure the pixel difference between the cardiac MRI modal image obtained after mode transformation and the original cardiac MRI image, and the registration deformation field is constrained by smoothing regularization loss. By using image patch contrast loss, the mutual information between input and output image patches is maximized, thus maintaining the consistency of anatomical structures. High-precision spatial alignment of CCTA images and cardiac MRI images is achieved through end-to-end collaborative optimization. Feature extraction and mapping training were performed on paired CCTA and echocardiogram images, specifically including: A three-dimensional cardiac model was built based on CCTA images, serving as a standard template for echocardiogram image registration. High-entropy frames are selected from cardiac ultrasound imaging sequences to ensure that the images contain clear anatomical features based on the principle of information entropy. A mapping relationship between the simulated signal of a three-dimensional cardiac model and the actual echocardiographic signal is established using a linear combination model or a polynomial nonlinear combination model, and a similarity metric LC is defined. 2 ; A global search strategy was used to locate the registration region, and local optimization was combined to achieve registration between dynamic cardiac ultrasound images and static CCTA images.

6. The method for extracting multimodal cardiac imaging features according to claim 5, characterized in that, The contrastive loss function is shown in the following equation: Among them, L PatchNCE To compare the loss function values, E x Let L be the expected value, and L be the number of levels to divide the image. To output image patch features, S l The number of image patches in the l-th level. For the corresponding input image patch features, For other input image patch features, l is the InfoNCE loss.

7. A cardiac multimodal image feature extraction device, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal cardiac imaging data, patient information, and case information in radiology and ultrasound departments of multiple centers; the multimodal cardiac imaging data includes cardiac ultrasound images, cardiac MRI images, and CCTA images; A multimodal data preprocessing module is used to label the multimodal cardiac imaging data and construct a multimodal imaging feature database. The labels on the multimodal cardiac imaging data are cardiac outline, intracavitary outline, myocarditis lesion area outline, and cardiac physiological and pathological structural features that can be reflected in each modality. The database platform construction module is used to build a database platform, filter and manage the uploaded multimodal cardiac imaging data; the database platform includes a data filtering and uploading terminal and a server storage control terminal; The communication link establishment module is used to receive the review request from the server storage control terminal and verify the identity of the requester, provide feedback to the approved requester and open the communication link, and display the data upload interaction interface on the requester's data filtering and uploading terminal for the requester to upload the patient's multimodal cardiac image data. The model training and optimization module is used to extract and map features of the corresponding cardiac structure-function based on labeled multimodal cardiac data and deep learning algorithms, train image modality conversion and registration models, and optimize model parameters. The image modality conversion and registration module is used to perform modality conversion and deformation on the input CCTA image data using the image modality conversion and registration model, so as to obtain the cardiac structural features of the corresponding cardiac ultrasound images and cardiac MRI images.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the cardiac multimodal image feature extraction method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the cardiac multimodal image feature extraction method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the cardiac multimodal image feature extraction method according to any one of claims 1-6.

Images