Myocardial infarction complication identification method based on heart multi-mode image

Abstract

The invention belongs to the technical field of medical image recognition, and particularly discloses a myocardial infarction complication recognition method based on a heart multi-modal image, and the method comprises the steps: firstly obtaining multi-modal medical image data and corresponding diagnosis report text data; constructing a myocardial infarction complication identification model; dividing the whole network into a feature extraction stage and an image text feature matching stage by adopting a separation training thought, carrying out back propagation on the network in the feature extraction stage through L1 loss, and optimizing weight parameters of the feature extraction network; performing back propagation on the network in the image text matching stage by comparing loss, and optimizing parameters of an image and a text encoder; and finally, collecting a medical image of the heart of a patient, inputting the medical image into the trained myocardial infarction complication identification model, and generating a character diagnosis report of heart disease complication. Therefore, early automatic diagnosis of myocardial infarction complications is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image recognition, and particularly relates to a method for identifying myocardial infarction complications based on multi-modal cardiac images. Background Art

[0002] Medical imaging is one of the most widely applied fields of multi-modal technology. Multi-modal medical imaging refers to obtaining image data of the same part or the same patient through multiple imaging techniques, so as to observe the tissue structure and lesion characteristics from different angles and provide more comprehensive diagnostic information. The multi-modal technology in medicine involves fusing different types of data, which helps doctors and experts obtain more comprehensive patient information to improve the accuracy and comprehensiveness of disease diagnosis, treatment planning, and efficacy evaluation. The multi-modal technology can integrate the information of multiple data sources, which helps to more comprehensively understand the disease characteristics and achieve precision medicine. Common types of multi-modal images include computed tomography, magnetic resonance imaging, positron emission tomography, ultrasound, etc. Each imaging mode has its own advantages and disadvantages in tissue imaging. Therefore, fusing the image data of different modalities can make full use of their complementary characteristics, thereby improving the effects of clinical diagnosis, treatment planning, and disease condition monitoring. Based on this, multi-modal medical imaging has received extensive attention in the computer field and the artificial intelligence field.

[0003] In addition, the mortality rate of myocardial infarction complications is as high as 60%, which is the most difficult and critical acute and severe cardiovascular disease. Applying artificial intelligence technology to accurately diagnose myocardial infarction complications at an early stage and reduce its mortality rate is a key and difficult issue in protecting life and health. However, due to the low quality of the labeled data of artificial intelligence technology, the weak causal reasoning ability of artificial intelligence, and the lack of close combination between engineering and clinical practice, artificial intelligence cannot be applied and implemented in the diagnosis of myocardial infarction complications. Automatically identifying myocardial infarction complications based on medical images is an urgent need in clinical work. There is an urgent need to develop an automatic medical image recognition tool for improving the diagnosis efficiency of myocardial infarction complications through the combination of medicine and engineering, which can effectively guarantee the emergency diagnosis and treatment process of major sudden disaster events and reduce the clinical mortality rate. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a method for identifying myocardial infarction complications based on multi-modal cardiac images to achieve early automatic diagnosis of myocardial infarction complications.

[0005] The method for identifying myocardial infarction complications based on multi-modal cardiac images of the present invention includes the following steps:

[0006] S1. Obtain multi-modal medical image data and corresponding diagnostic report text data;

[0007] S2. Construct a myocardial infarction complication recognition model, which includes a multi-modal feature extraction module, a multi-modal feature fusion module, and an image-text pairing module. The multi-modal feature extraction module encodes different modal cardiac images input, and performs feature complementarity to obtain refined features of different modal features; the multi-modal feature fusion module fuses different features of different modalities to obtain aggregated features; the image-text pairing module learns the relationship between the text diagnosis report and the fused features through the input of the text diagnosis report of cardiac disease complications and the obtained fused features.

[0008] S3. Adopting the idea of separate training, divide the overall network into a feature extraction stage and an image-text feature matching stage. Perform backpropagation on the network in the feature extraction stage through the L1 loss to optimize the weight parameters of the feature extraction network; perform backpropagation on the network in the image-text matching stage through the contrastive loss to optimize the parameters of the image and text encoders.

[0009] S4. Collect medical images of the patient's heart, input them into the trained myocardial infarction complication recognition model, and use the model to process and analyze the input heart images to generate a text diagnosis report of cardiac disease complications.

[0010] Preferably, in S1, M groups of cardiac medical images with N modalities and M corresponding diagnostic report texts are collected, where the input medical images are Unify the channel dimension of the multi-modal image data to 1 through 1*1 convolution, and collect the corresponding diagnostic report texts Use a text encoder with a Transformer architecture to extract features from the input text to obtain a text feature vector

[0011] Preferably, the multi-modal feature extraction module in S2 includes a convolutional layer, a prototype selection layer, a correlation calculation layer, and a feature complementarity layer. The convolutional layer is used to extract the initial feature f of each modality n , the prototype selection layer is used to generate prototypes through cosine similarity, the correlation calculation layer is used to calculate the correlation between the prototype feature f prototype and each initial feature f n , and the feature complementarity layer is used to perform information exchange of features between modalities to obtain refined modal features f' n ; the specific processing steps of the multi-modal feature extraction module are as follows:

[0012] S2.1. Convolutional layer, extract the initial feature f of each modality n ;

[0013] S2.2. Prototype selection layer, compare each feature f n extracted by the convolutional layer with the text feature ftext Calculate the similarity, and take the feature corresponding to the maximum similarity as the prototype feature. The formula is as follows

[0014] Take f prototype Satisfy

[0015] S2.3. Correlation calculation layer, which calculates the similarity between the prototype feature f prototype and the features f n of each modality. The formula is as follows

[0016]

[0017] Among them, represents L2 normalization, represents matrix multiplication, and S n represents the similarity between the initial feature of the nth modality and the fused feature;

[0018] S2.4. Feature complementary layer, which effectively transmits information between modalities using the cross-attention mechanism to refine the feature f′ n obtained from the following formula

[0019]

[0020] Among them, σ K and σ V respectively represent the fully connected networks for calculating keys and values, f′ n is the refined feature of the nth modality, represents matrix multiplication.

[0021] Preferably, in the multi-modal feature fusion module in S2, the input refined feature f′ n is obtained by concatenation in the channel dimension to get

[0022] The channel attention mechanism is adopted to obtain the aggregated feature The formula is as follows, f fusion = f concat ·Softmax(σ(W 1 ·Avg(f concat )) + b 1 ) + σ(W 2 ·Max(f concat )) + b 2 ))

[0023] Among them, f concat is the concatenation of f n in the channel dimension. Avg represents average pooling, Max represents max pooling, σ is the activation function, W 1 and W 2are learnable weights, b 1 and b 2 are learnable biases.

[0024] Preferably, in S2, the image-text pairing module calculates the cosine similarity between the aggregated feature vector f fusion and the text feature vector f text as follows.

[0025]

[0026] i and j represent the cosine similarity between the aggregated feature of the i-th pair of medical images and the text feature of the j-th diagnostic report.

[0027] Preferably, in S3, the weight parameters in the feature extraction stage are updated by the L1 loss, as follows.

[0028]

[0029] where the distance between the refined feature after complementation and the prototype feature is calculated by the L1 loss, and k represents the number of times of parameter update.

[0030] Preferably, in S3, the contrastive loss is used to train the model in the image-text feature matching stage. The specific steps are as follows:

[0031] Train the image-text pairing module with the following formula using the contrastive loss. Assuming that M groups of cardiac medical images and M corresponding diagnostic report texts are input, an M×M similarity matrix can be obtained, and the similarity on the diagonal is the largest.

[0032] Image-to-text direction loss

[0033] Text-to-image direction loss

[0034] Loss

[0035] The present invention also provides a myocardial infarction complication recognition system based on cardiac multimodal images, including:

[0036] A data acquisition unit for acquiring multimodal cardiac medical images of patients;

[0037] A model construction unit for constructing a myocardial infarction complication recognition model, which includes a multimodal feature extraction module, a multimodal feature fusion module, and an image-text pairing module;

[0038] The data collection and training unit collects multimodal medical image data and the corresponding diagnostic report text data. The idea of ​​separate training is adopted to divide the overall network into the feature extraction stage and the image-text feature matching stage. The network in the feature extraction stage is back-propagated through L1 loss to optimize the weight parameters of the feature extraction network so that the model can extract better refined features. The network in the image-text matching stage is back-propagated through contrast loss to optimize the parameters of the image and text encoders so that the model gradually learns how to bring the aggregate features and text features closer in the embedding space.

[0039] The disease recognition unit collects medical images of the patient's heart and inputs them into a trained myocardial infarction complication recognition model. The model is used to process and analyze the input heart images to generate a text diagnosis report of heart disease complications based on the heart images.

[0040] Finally, the present invention also provides a computer-readable storage medium, on which is stored the above-mentioned myocardial infarction complication identification system based on cardiac multimodal imaging, and when the program is executed by a processor, myocardial infarction complication identification based on multimodal cardiac imaging is realized.

[0041] The beneficial effects of the present invention are:

[0042] Compared with the prior art, firstly, the present invention makes full use of multimodal medical imaging information, considers the relationship between each modality, and complementarily refines their features, so that the fused image features have the advantageous information of each modality. Secondly, the invention adopts the idea of ​​separation training in the training process, which effectively avoids the problems of high training difficulty, difficult back propagation calculation, high computing resource requirements, etc., and maintains the connection between the two stages of training while achieving the above-mentioned beneficial effects, ensuring the accuracy of the training. Finally, the present invention also considers the distance between the image aggregation features and the diagnostic report text features in the embedding space, and realizes the process from inputting multimodal cardiac medical images to outputting the diagnostic report text of myocardial infarction complications. Thereby, early automatic diagnosis of myocardial infarction complications can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 is a schematic flow chart of the method for identifying complications of myocardial infarction of the present invention;

[0045] Figure 2It is a schematic diagram of the network structure of the myocardial infarction complication recognition model in the embodiment.

[0046] Figure 3 It is a schematic diagram of the training process of the myocardial infarction complication recognition model in the embodiment. Detailed implementation manners

[0047] The following further describes the detailed implementation manners of the present invention. It should be noted here that the description of these implementation manners is for helping to understand the present invention, but does not constitute a limitation to the present invention. In addition, the technical features involved in the various implementation manners of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] Embodiment 1

[0049] In this embodiment, a method for recognizing myocardial infarction complications based on multi-modal cardiac images is proposed. As Figure 1 shown, the specific steps are as follows:

[0050] S1. Obtain multi-modal medical image data and corresponding diagnostic report text data. To specifically illustrate the implementation process of the present invention, the input data is specified here. Collect N groups of cardiac medical images and N corresponding diagnostic report texts. The cardiac medical images include echocardiogram data cardiac MRI data cardiac CT data Collect the corresponding diagnostic report texts

[0051] S2. Build a model for recognizing myocardial infarction complications based on multi-modal cardiac images. The model is as Figure 2 shown, and includes a multi-modal feature extraction module, a multi-modal feature fusion module, and an image-text pairing module. Among them, the multi-modal feature extraction module encodes the input cardiac images of different modalities and performs feature complementarity on them to obtain refined features of different modality features; the multi-modal feature fusion module fuses different features of different modalities to obtain aggregated features; the image-text pairing module inputs the text diagnosis report of myocardial infarction complications based on expert experience and the fused features obtained in the above steps, and learns the relationship between the text diagnosis report and the fused features from them.

[0052] The steps of the multi-modal feature extraction module include:

[0053] S2.1. Preliminary feature extraction. Input the medical image data input 1 、input 2 、input 3 to a 1×1 convolution to unify the channel dimension of the modality data to 1, and obtain the initial feature f 1 、f2 , Extract features from the corresponding diagnostic report text collected in the above steps through the text encoder of the Transformer architecture to obtain text feature vectors

[0054] S2.2, Prototype selection. Calculate the similarity between each feature f extracted by the convolutional layer n and the text feature f text Perform similarity calculation. Take the feature corresponding to the maximum similarity as the prototype feature. Since there is a maximum cosine similarity between the prototype feature and the corresponding text feature, that is, there is a maximum similarity between the prototype feature and the text feature, so it is used as the benchmark for feature refinement. The formula is as follows

[0055] Take f prototype Satisfy

[0056] S2.3, Correlation calculation. Calculate the similarity between the prototype feature f prototype and the features f of each modality n The similarity calculation formula is as follows

[0057]

[0058] Indicates L2 normalization Indicates matrix multiplication, S n Indicates the similarity between the initial feature of the nth modality and the fused feature. Here, n = 1, 2, 3

[0059] S2.4, Feature complementation. Use the cross-attention mechanism to effectively transmit information between modalities. The image information of different modalities contains different information about myocardial infarction complications. The information here has advantages and disadvantages. Therefore, the cross-attention mechanism is used to exchange information between each modality to achieve the purpose of information complementation, refine the initial features of each modality, and make it have the information of other modalities. The refined feature f′ n Is obtained from the following formula

[0060]

[0061] Among them, σ K and σ V Respectively represent the fully connected networks used to calculate the key and value. f′ n Is the refined feature of the nth modality Indicates matrix multiplication. Here, n = 1, 2, 3

[0062] The multi-modal feature fusion module includes the following steps

[0063] Input the refined features f′ 1 、f′2 , f′ 3 , obtained by concatenation in the channel dimension Adopt a channel attention mechanism, which is used here as the image encoder of the disease recognition module to obtain aggregated features The formula is as follows

[0064] f fusion = f concat ·Softmax(σ(W 1 ·Avg(f concat ) + b 1 ) + σ(W 2 ·Max(f concat ) + b 2 ))

[0065] Among them, f concat is the concatenation of f 1 , f 2 , f 3 in the channel dimension. Avg represents average pooling, Max represents max pooling, σ is the activation function, W 1 and W 2 are learnable weights, and b 1 and b 2 are learnable biases

[0066] The image-text pairing module includes the following steps

[0067] Calculate the cosine similarity between the aggregated feature vector f fusion and the text feature vector f text . i and j represent the cosine similarity between the aggregated feature of the i-th pair of medical images and the text feature of the j-th diagnostic report

[0068]

[0069] S3. Training steps, as shown in Figure 3 , adopt the idea of separate training, which greatly improves the efficiency of network training. Moreover, due to the update of the parameter weights of the text encoder in the second stage, that is, the text features are changed, which also affects the training of the network in the first stage to a certain extent, ensuring that the training of the two stages maintains a certain connection

[0070] The weight parameters of the feature complementary layer stage network are updated through the L1 loss. The formula is as follows

[0071]

[0072] Among them, the distance between the refined features and the prototype features after complementation is calculated by L1 loss, and n represents the three refined features obtained by the feature complementation layer. In addition, since the weights of the text encoder are updated in the training of the subsequent steps, and it is easy to obtain the close relationship between the prototype features and the text features according to the above steps, the content of the subsequent step training is also implicitly included here, based on which the weights of the feature complementation layer network are updated.

[0073] In the image-text matching phase, the network performs back propagation through contrast loss to optimize the parameters of the image and text encoders, so that the model gradually learns how to bring the aggregate features and text features closer in the embedding space. The formula is as follows:

[0074] Image to text directional loss,

[0075] Text-to-image directional loss,

[0076] loss,

[0077] S4. Collect medical images of the patient's heart, such as MRI images, echocardiograms, etc., and input them into a trained multimodal cardiac imaging disease automatic recognition model. Use the model to process and analyze the input cardiac images to generate a diagnostic text report of myocardial infarction complications based on cardiac images.

[0078] Example 2

[0079] Based on Example 1, in this embodiment, a myocardial infarction complication recognition system based on cardiac multimodal imaging is proposed, including:

[0080] A data acquisition unit, used for acquiring multimodal cardiac medical images of patients;

[0081] Model construction unit: constructs a model for disease recognition of multimodal cardiac images, the model includes a multimodal feature extraction module, a multimodal feature fusion module, and an image-text pairing module. The multimodal feature extraction module encodes the input cardiac images of different modalities and performs feature complementation on them, thereby correcting and obtaining different features of different modalities; the multimodal feature fusion module fuses different features of different modalities to obtain fused features; the disease recognition module uses the text diagnosis report of heart disease complications based on expert experience and the fused features obtained in the above steps as input, and learns the relationship between the text diagnosis report and the fused features;

[0082] Data collection and training unit: Collect multimodal medical image data and corresponding diagnostic report text data, adopt the idea of ​​separate training, divide the overall network into feature extraction stage and image-text feature matching stage, back-propagate the feature extraction stage network through L1 loss, optimize the weight parameters of the feature extraction network, so that the model can extract better refined features. Back-propagate the image-text matching stage network through contrast loss, optimize the parameters of the image and text encoders, so that the model gradually learns how to bring the aggregate features and text features closer in the embedding space;

[0083] Disease identification unit: collects medical images of the patient's heart, such as MRI images, echocardiograms, etc., and inputs them into a trained multimodal cardiac imaging disease automatic identification model. The model is used to process and analyze the input cardiac images to generate a diagnostic report of cardiac disease complications based on the cardiac images, which is presented in text form.

[0084] Example 3

[0085] Based on the above embodiments, in this embodiment, a computer-readable storage medium is provided, on which the above-mentioned myocardial infarction complication identification system based on cardiac multimodal imaging is stored. When the program is executed by a processor, myocardial infarction complication identification based on multimodal cardiac imaging is realized.

Claims

1. A method for identifying complications of myocardial infarction based on cardiac multimodal imaging, characterized in that: The following steps are involved: S1. Acquire multimodal medical image data and corresponding diagnostic report text data; S2. Constructing a myocardial infarction complication recognition model, the model includes a multimodal feature extraction module, a multimodal feature fusion module, and an image-text pairing module. The multimodal feature extraction module encodes input cardiac images of different modalities and performs feature complementation to obtain refined features of features of different modalities; the multimodal feature fusion module fuses different features of different modalities to obtain aggregate features; The image-text pairing module learns the relationship between the text diagnosis report and the fusion features through the input of the text diagnosis report of the complications of heart disease and the acquired fusion features; S3. Adopting the idea of ​​separate training, the overall network is divided into the feature extraction stage and the image-text feature matching stage. The feature extraction stage network is back-propagated through L1 loss to optimize the weight parameters of the feature extraction network; the image-text matching stage network is back-propagated through contrast loss to optimize the parameters of the image and text encoders; S4. Collect medical images of the patient's heart and input them into a trained myocardial infarction complication recognition model. Use the model to process and analyze the input heart images to generate a text diagnosis report of heart disease complications.

2. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: In S1, M groups of cardiac medical images with N modalities and M corresponding diagnostic report texts are collected, where the input medical image is The channel dimension of multimodal imaging data is unified to 1 through 1*1 convolution, and the corresponding diagnostic report text is collected The text encoder using the Transformer architecture extracts features from the input text and obtains a text feature vector 3. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: The multimodal feature extraction module in S2 includes a convolutional layer, a prototype selection layer, a correlation calculation layer, and a feature complementation layer. The convolutional layer is used to extract the initial features f of each modality. n The prototype selection layer is used to generate prototypes through cosine similarity, and the correlation calculation layer is used to calculate the prototype feature f prototype With each initial feature f n The feature complementation layer is used to exchange information between modal features to obtain the refined modal feature f′ n ; The specific processing steps of the multimodal feature extraction module are as follows: S2.1, convolutional layer, extracts the initial features f of each modality n ; S2.2, prototype selection layer, each feature f extracted by the convolutional layer n With the text feature f text Perform similarity calculation and take the feature corresponding to the maximum similarity as the prototype feature. The formula is as follows Take f prototype satisfy S2.3, correlation calculation layer, the prototype feature f prototype and the characteristics f of each mode n To calculate the similarity, the formula is as follows: in, represents L2 normalization, represents matrix multiplication, S n Represents the similarity between the initial features and fusion features of the nth modality; S2.4, feature complementation layer, uses the cross-attention mechanism to effectively transfer information between modalities and refine the feature f′ n From the following formula, Among them, σ K and σ V Denote the fully connected networks used to calculate keys and values, respectively, and f′ n is the refined feature of the nth mode, Represents matrix multiplication.

4. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: The multimodal feature fusion module in S2 inputs the refined feature f′ n , obtained by concatenating in the channel dimension Adopt channel attention mechanism to obtain aggregate features The formula is as follows, f fusion =f concat ·Softmax(σ(W1·Avg(f concat )+b1)+σ(W2·Max(f concat )+b2)) Among them, f concat f n In the channel dimension, Avg represents average pooling, Max represents maximum pooling, σ is the activation function, W1 and W2 are learnable weights, and b1 and b2 are learnable biases.

5. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: The image-text pairing module in S2 calculates the aggregate feature vector f fusion and the text feature vector f text The cosine similarity between is as follows: i and j represent the cosine similarity between the aggregated features of the i-th medical impact and the j-th diagnostic report text features.

6. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: The weight parameters of the feature extraction stage in S3 are updated through L1 loss, and the formula is as follows: Among them, the distance between the complementary refined features and the prototype features is calculated by L1 loss, and k represents the number of parameter updates.

7. The method for identifying complications of myocardial infarction based on cardiac multimodal imaging according to claim 1, characterized in that: In the image-text feature matching stage in S3, contrast loss is used to train the model. The specific steps are as follows: The image-text pairing module is trained using the following formula, using contrast loss. Assuming that M sets of cardiac medical images and M corresponding diagnosis report texts are input, an M*M similarity matrix can be obtained, and the similarity on the diagonal is the largest; Image to text directional loss, Text-to-image directional loss, loss, 8. A myocardial infarction complication recognition system based on cardiac multimodal imaging, characterized in that: include: A data acquisition unit, used for acquiring multimodal cardiac medical images of patients; A model building unit, which builds a myocardial infarction complication recognition model, wherein the model includes a multimodal feature extraction module, a multimodal feature fusion module, and an image-text pairing module; The data collection and training unit collects multimodal medical image data and the corresponding diagnostic report text data. The idea of ​​separate training is adopted to divide the overall network into the feature extraction stage and the image-text feature matching stage. The network in the feature extraction stage is back-propagated through L1 loss to optimize the weight parameters of the feature extraction network so that the model can extract better refined features. The network in the image-text matching stage is back-propagated through contrast loss to optimize the parameters of the image and text encoders so that the model gradually learns how to bring the aggregate features and text features closer in the embedding space. The disease recognition unit collects medical images of the patient's heart and inputs them into a trained myocardial infarction complication recognition model. The model is used to process and analyze the input heart images to generate a text diagnosis report of heart disease complications based on the heart images.

9. A computer-readable storage medium storing the myocardial infarction complication identification system based on cardiac multimodal imaging according to claim 8, characterized in that: The storage medium stores the above-mentioned myocardial infarction complication identification system based on cardiac multimodal images, and when the program is executed by the processor, myocardial infarction complication identification based on multimodal cardiac images is realized.

Images