Coronary heart disease prediction system based on multimodal carotid artery data
Feature extraction and fusion of multimodal carotid artery data through deep learning methods, solving the problems of data imbalance and insufficient text data utilization in coronary heart disease prediction, realizing accurate prediction of multimodal data, and improving the accuracy of the model and efficiency.
Patent Information
- Application Number
- CN202210032919.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-01-12
AI Technical Summary
The prior art has problems with uneven data set number and distribution in the prediction of coronary heart disease, and multimodal data is not fully utilized, and medical text data is difficult to be directly used for deep network computing, affecting the accuracy of the model.
A multimodal carotid artery data prediction system based on deep learning is adopted to obtain image and text data, extract image and text features, and perform cross-modal fusion, and input it into the trained coronary heart disease classification model for prediction.
Accurate prediction of multimodal carotid artery data is achieved, the accuracy and efficiency of coronary heart disease prediction is improved, and the performance of the model is enhanced using multimodal data.
Smart Images

Figure CN114464326B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing and multimodal data fusion, and in particular to a coronary heart disease prediction system based on multimodal carotid artery data. Background Art
[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.
[0003] Cardiovascular diseases have a high incidence and mortality rate, and the main mechanism is atherosclerosis. Studies have shown that carotid intima-media thickness, carotid plaques, clinical examination indicators, etc. are all related to the occurrence of cardiovascular events and coronary heart disease. Currently, existing studies have analyzed the relationship between single-modal data and coronary heart disease, and have not fully integrated and utilized multimodal data.
[0004] In recent years, deep learning has continued to gain momentum. However, there are still several problems in predicting coronary heart disease using deep learning methods:
[0005] (1) The problem of imbalance in the number and distribution of data sets. Medical data requires a certain amount of manpower and material resources to collect, which takes a long time, and the distribution of positive and negative samples is uneven, which will greatly affect the training process and output results of the deep learning model.
[0006] (2) The medical data generated by relevant examinations usually exist in a multimodal form. Currently, existing studies all analyze the relationship between a certain type of single-modal data and CHD, and have not fully integrated and utilized multimodal data to predict CHD.
[0007] (3) Medical text data cannot be directly used for deep network calculations and needs to be digitally represented. For long medical texts, simple CNNs, word vector generation models (CBOM, skip-grams), long short-term memory networks, etc. cannot well represent medical entities in vectors, affecting the accuracy of the model. Summary of the invention
[0008] In order to address the deficiencies of the prior art, the present invention provides a coronary heart disease prediction system based on multimodal carotid artery data; a deep learning method is used to effectively mine multimodal carotid artery data to predict coronary heart disease.
[0009] In a first aspect, the present invention provides a coronary heart disease prediction system based on multimodal carotid artery data;
[0010] The coronary heart disease prediction system based on multimodal carotid artery data includes:
[0011] An acquisition module is configured to: acquire image data and text data of a patient to be predicted; wherein the image data refers to images of carotid arteries in different modalities;
[0012] A feature extraction module is configured to: extract image features from images of each modality;
[0013] A text feature extraction module is configured to: extract text features from text data;
[0014] A cross-modal fusion module is configured to: fuse image features of different modalities to obtain fused image features;
[0015] The prediction module is configured to: concatenate text features with fused image features, input the concatenated features into the trained coronary heart disease classification model, and obtain a coronary heart disease prediction result.
[0016] In a second aspect, the present invention further provides an electronic device, comprising:
[0017] a memory for non-transitory storage of computer-readable instructions; and
[0018] a processor for executing the computer readable instructions,
[0019] When the computer readable instructions are executed by the processor, the following steps are performed:
[0020] Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities;
[0021] Extract image features for each modality of image;
[0022] Extract text features from text data;
[0023] Perform feature fusion on image features of different modalities to obtain fused image features;
[0024] The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
[0025] In a third aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the following steps are performed:
[0026] Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities;
[0027] Extract image features for each modality of image;
[0028] Extract text features from text data;
[0029] Perform feature fusion on image features of different modalities to obtain fused image features;
[0030] The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
[0031] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program, wherein the computer program is configured to perform the following steps when running on one or more processors:
[0032] Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities;
[0033] Extract image features for each modality of image;
[0034] Extract text features from text data;
[0035] Perform feature fusion on image features of different modalities to obtain fused image features;
[0036] The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] The present invention achieves accurate prediction of coronary heart disease by fusing multiple types of image data, fusing image data with text data, and utilizing the fused multimodal carotid artery data. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0040] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION
[0041] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0042] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0043] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0044] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.
[0045] Embodiment 1
[0046] This embodiment provides a coronary heart disease prediction system based on multimodal carotid artery data;
[0047] like Figure 1 As shown, the coronary heart disease prediction system based on multimodal carotid artery data includes:
[0048] An acquisition module is configured to: acquire image data and text data of a patient to be predicted; wherein the image data refers to images of carotid arteries in different modalities;
[0049] A feature extraction module is configured to: extract image features from images of each modality;
[0050] A text feature extraction module is configured to: extract text features from text data;
[0051] A cross-modal fusion module is configured to: fuse image features of different modalities to obtain fused image features;
[0052] The prediction module is configured to: concatenate text features with fused image features, input the concatenated features into the trained coronary heart disease classification model, and obtain a coronary heart disease prediction result.
[0053] Furthermore, the carotid artery images of different modalities include: a carotid artery CT image and a carotid artery MR image.
[0054] For example, multimodal carotid artery data is fused to predict coronary heart disease through deep learning methods. The relevant examinations generate multimodal medical data, including digital measurements (examination data), verbal descriptions (symptoms and notes, vital signs, etc.), and image data (ultrasound, CT and MR scans, etc.).
[0055] Furthermore, the text data includes: digital test results and symptom description language.
[0056] Furthermore, the image features are extracted from the image of each modality, specifically by using a feature extraction network; wherein,
[0057] The feature extraction network includes: a Resnet101 backbone network, a channel attention mechanism layer and a convolution layer connected in sequence.
[0058] It should be understood that image feature extraction uses Resnet101 as the backbone network for extracting image features, and then passes through a channel attention mechanism and a two-dimensional convolution to make the network focus on more meaningful areas in the image.
[0059] Furthermore, the extraction of text features from text data is specifically implemented through a BERT encoder.
[0060] For example, text data feature extraction uses stacked Transformer encoder modules to effectively vectorize unstructured text data, which can effectively capture the rich semantic relationships contained in the preceding and following word order of long texts, and thus correctly represent medical entities.
[0061] Text feature extraction is used to extract features from medical text data. The core of the model consists of BERT Encoder, which consists of multiple BERT Layers. Each BERT Layer is actually an Encoder Block in Transformer. Each encoder layer consists of two layers, a self-attention mechanism layer and a feedforward neural network layer.
[0062] Furthermore, the image features of different modalities are fused to obtain fused image features; the specific feature fusion method used is: weighted fusion.
[0063] Furthermore, in the weighted fusion, it is:
[0064] First, calculate the query vector Query and key vector Key of the MR image;
[0065] Then calculate the value vector Value of the CT image;
[0066] Calculate the similarity between Query and Key;
[0067] After the similarity is probabilized, it is used as the weight to perform weighted summation on the Value of the CT image to obtain the feature fusion result.
[0068] Furthermore, the query vector Query is obtained by multiplying the feature map of the MR image by the weight matrix of the query vector.
[0069] Furthermore, the key vector Key is obtained by multiplying the feature map of the MR image by the weight matrix of the key vector.
[0070] Furthermore, the value vector Value is obtained by multiplying the feature map of the CT image by the weight matrix of the value vector.
[0071] Furthermore, the similarity is probabilized by using a softmax function.
[0072] Multimodal image feature fusion uses cross-modal Transformer to fuse multimodal image information. For the MR and CT image data of the carotid artery, the cross-modal Transformer fusion calculates the query and key of the MR image, calculates the value of the CT image, calculates the similarity between the query and the key, and then probabilizes it and uses it as a weight on the value of the CT image.
[0073] Furthermore, the training process of the trained coronary heart disease classification model includes:
[0074] Build a classification model;
[0075] Constructing a training set, wherein the training set is carotid artery image data and text data with known coronary heart disease classification labels;
[0076] The training set is input into the classification model, and the classification model is trained to obtain a trained coronary heart disease classification model.
[0077] Furthermore, after the acquisition module and before the feature extraction module, it also includes: a preprocessing module;
[0078] The preprocessing module is configured to preprocess the acquired image data and text data.
[0079] Furthermore, preprocessing the image data refers to rotating and mirroring the image to increase the number of data sets.
[0080] Preprocessing the text data refers to oversampling the text data to increase the number of data sets.
[0081] The image data is enhanced by rotation, mirroring, etc., and the medical text vectors are analyzed using oversampling technology to generate new data to increase the number of data sets.
[0082] Embodiment 2
[0083] This embodiment further provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device performs the following steps:
[0084] Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities;
[0085] Extract image features for each modality of image;
[0086] Extract text features from text data;
[0087] Perform feature fusion on image features of different modalities to obtain fused image features;
[0088] The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
[0089] Embodiment 3
[0090] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the following steps are performed:
[0091] Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities;
[0092] Extract image features for each modality of image;
[0093] Extract text features from text data;
[0094] Perform feature fusion on image features of different modalities to obtain fused image features;
[0095] The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. Coronary heart disease prediction system based on multimodal carotid artery data, Its characteristics are: include: An acquisition module is configured to: acquire image data and text data of a patient to be predicted; wherein the image data refers to images of carotid arteries in different modalities; the images of carotid arteries in different modalities include: carotid artery CT images and carotid artery MR images; the text data includes: digital test results and symptom description language; A preprocessing module is configured to: preprocess the acquired image data and text data; wherein preprocessing the image data refers to rotating and mirroring the image to increase the number of data sets; preprocessing the text data refers to oversampling the text data to increase the number of data sets; A feature extraction module is configured to: extract image features from images of each modality; specifically, extract features through a feature extraction network; wherein the feature extraction network includes: a Resnet101 backbone network, a channel attention mechanism layer, and a convolution layer connected in sequence; Resnet101 is used as the backbone network for extracting image features, and a channel attention mechanism and a two-dimensional convolution are used to make the network focus on more meaningful areas in the image; The text feature extraction module is configured to: extract text features from text data; effectively vectorize unstructured text data through stacked Transformer encoder modules, effectively capture the rich semantic relationships contained in the sequence of words in long texts, and correctly represent medical entities; the core is composed of BERT Encoder, which is composed of multiple layers of BERT Layer, and each BERT Layer is an Encoder Block in Transformer; each encoder layer contains two layers, a self-attention mechanism layer and a feedforward neural network layer; The cross-modal fusion module is configured to: fuse the image features of different modalities to obtain fused image features; the specific feature fusion method used is: weighted fusion; in the weighted fusion, the query vector Query and the key vector Key of the MR image are first calculated; then the value vector Value of the CT image is calculated; the similarity between the Query and the Key is calculated; after the similarity is probabilized, the Value of the CT image is weightedly summed up as a weight to obtain the feature fusion result; The query vector Query or key vector Key is obtained by multiplying the feature map of the MR image with the weight matrix of the query vector or key vector; the value vector Value is obtained by multiplying the feature map of the CT image with the weight matrix of the value vector; the probabilization of the similarity is specifically performed by using a softmax function for probabilization; A prediction module is configured to: concatenate text features with fused image features, input the concatenated features into a trained coronary heart disease classification model, obtain a coronary heart disease prediction result, and achieve accurate prediction of coronary heart disease; The trained coronary heart disease classification model; the training process includes: Build a classification model; Constructing a training set, wherein the training set is carotid artery image data and text data with known coronary heart disease classification labels; The training set is input into the classification model, and the classification model is trained to obtain a trained coronary heart disease classification model.
2. The coronary heart disease prediction system based on multimodal carotid artery data as claimed in claim 1, Its characteristics are: The extraction of text features from text data is specifically implemented through a BERT encoder.
3. An electronic device, suitable for the coronary heart disease prediction system based on multimodal carotid artery data according to claim 1, Its characteristics are: include: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer readable instructions are executed by the processor, the following steps are performed: Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities; Extract image features for each modality of image; Extract text features from text data; Perform feature fusion on image features of different modalities to obtain fused image features; The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
4. A storage medium, suitable for the coronary heart disease prediction system based on multimodal carotid artery data according to claim 1, Its characteristics are: Non-transitory storage of computer readable instructions, wherein when the non-transitory computer readable instructions are executed by a computer, the following steps are performed: Acquire image data and text data of the patient to be predicted; wherein the image data refers to images of the carotid artery in different modalities; Extract image features for each modality of image; Extract text features from text data; Perform feature fusion on image features of different modalities to obtain fused image features; The text features are concatenated with the fused image features, and the concatenated features are input into the trained coronary heart disease classification model to obtain the coronary heart disease prediction results.
