An auxiliary method for heart failure based on multi-modal data fusion

Abstract

The present application relates to the technical field of multi-modal data fusion, and particularly relates to an auxiliary judgment method for heart failure based on multi-modal data fusion, which firstly constructs image-text pair data of all data, then constructs the architecture of a joint model, and then respectively trains an image encoder and a parameterized text encoder based on a ranking standard to learn the features of images and texts, and then uses two fully connected layers (with the same neural network structure) on the joint embedding space to respectively evaluate the severity of edema from images and reports, which is used as a reference for doctors, thereby further improving the efficiency of doctors in diagnosing patients with acute congestive heart failure.

Description

Technical Field

[0001] This invention relates to the field of multimodal data fusion technology, and specifically to an auxiliary method for the diagnosis of heart failure based on multimodal data fusion. Background Technology

[0002] Acute heart failure, also known as acute congestive heart failure (CHF), refers to a significant and rapid decrease in cardiac output caused by acute cardiac events, leading to insufficient tissue and organ perfusion and acute congestion syndrome. Common causes include acute exacerbations of chronic heart failure, acute hemodynamic instability, and right ventricular infarction. Acute left ventricular failure is the most common, mainly manifested by severe dyspnea, forced sitting posture, and pale complexion. Pulmonary edema is a common complication in patients with grade III acute heart failure, but doctors often struggle to accurately assess its severity during examination, leading to inaccurate medication prescriptions. The emergence of multimodal approaches has rapidly advanced the use of medical data fusion to improve the accuracy of medical diagnostic models, saving doctors considerable time previously spent assessing conditions and improving their efficiency. While numerous studies have explored the use of data fusion to improve model quality, research on its application in assessing acute congestive heart failure remains limited.

[0003] Data fusion is the process of combining data from multiple sources—sensor inputs, information processing modules, databases, or knowledge bases—into a unified representation format. Data fusion architectures have undergone various developmental stages and have gradually evolved into two techniques: rule-based decision-making and fuzzy logic decision-making. Multimodal data fusion can improve detection in a variety of ways. In the past, data fusion has been proposed and developed to improve cardiac detection. In this work, image and text data are fused with a classification cluster-based concept to improve the accuracy of physicians' clinical assessment of patients with acute heart failure. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for assisting in the diagnosis of heart failure based on multimodal data fusion, which assesses the severity of edema from images and reports, thereby further improving the diagnostic efficiency of physicians for patients with acute congestive heart failure.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for auxiliary diagnosis of heart failure based on multimodal data fusion, including...

[0007] Preprocessing: Preprocess the chest X-ray and provide the text of the radiology report to the tokenizer;

[0008] The joint model architecture is built after processing the input data, including the joint embedding loss. Associate the image embedding I with the text embedding R in the joint embedding space;

[0009] Joint representation learning applies ranking-based criteria to... and A parameterized image encoder is trained to learn image and text feature representations. and Given an image-text pair Randomly select a replacement image from X. A substitute report Map s(j) generates a random permutation of {1,2,...,N}; matching pairs (I j ,R j The feature representation between (I) pairs is greater than that between mismatched pairs (I) pairs. s(j) ,R j ) and (I j ,R s(j) Features between them are represented as closer;

[0010] The network construction and classification adopt the loss based on stochastic gradient minimization in Equation 2 and learn using an adaptive learning rate scheduler; the dot product is used as its similarity measure in Equation 1; the dataset is divided into training and test sets, and all hyperparameters are selected based on the results of 5x cross-validation in the training set;

[0011] Formula 1 is:

[0012] Formula 2 is:

[0013]

[0014] After determining the severity of edema, the condition of patients with acute congestive heart failure is linearly reflected.

[0015] The beneficial effects of this invention are as follows: First, this invention constructs image-text pairs of all data, then builds the architecture of a joint model, and then applies a ranking-based standard to train the image encoder and the parameterized text encoder to learn the features of the image and text respectively. Then, two fully connected layers (with the same neural network structure) are used in the joint embedding space to assess the severity of edema from the image and the report respectively. This severity serves as a reference for physicians, thereby further improving the efficiency of physicians in diagnosing patients with acute congestive heart failure. Attached Figure Description

[0016] Figure 1This is a joint model architecture diagram of an auxiliary judgment method for heart failure based on multimodal data fusion, which is a specific embodiment of the present invention.

[0017] Figure 2 The flowchart below shows a CHF condition auxiliary assessment method based on multimodal data fusion, which is a specific embodiment of the present invention. Detailed Implementation

[0018] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0019] Please refer to Figure 1 as well as Figure 2 A method for auxiliary diagnosis of heart failure based on multimodal data fusion, including...

[0020] Preprocessing: Preprocess the chest X-ray and provide the text of the radiology report to the tokenizer;

[0021] The joint model architecture is built after processing the input data, including the joint embedding loss. Associate the image embedding I with the text embedding R in the joint embedding space;

[0022] Joint representation learning applies ranking-based criteria to... and A parameterized image encoder is trained to learn image and text feature representations. and Given an image-text pair Randomly select a replacement image from X. A substitute report Map s(j) generates a random permutation of {1,2,...,N}; matching pairs (I j ,R j The feature representation between (I) pairs is greater than that between mismatched pairs (I) pairs. s(j) ,R j ) and (I j ,R s(j) Features between them are represented as closer;

[0023] The network construction and classification adopt the loss based on stochastic gradient minimization in Equation 2 and learn using an adaptive learning rate scheduler; the dot product is used as its similarity measure in Equation 1; the dataset is divided into training and test sets, and all hyperparameters are selected based on the results of 5x cross-validation in the training set;

[0024] Formula 1 is:

[0025]

[0026] Formula 2 is:

[0027]

[0028] After determining the severity of edema, the condition of patients with acute congestive heart failure is linearly reflected.

[0029] Furthermore, the chest X-ray image is randomly flipped and rotated during training and cropped to 2048×2048 pixels.

[0030] Furthermore, the chest X-ray maintains the original image resolution during preprocessing.

[0031] As described above, by maintaining the original image resolution, subtle differences in images between different degrees of pulmonary edema severity can be captured.

[0032] Furthermore, the radiology report extracts the impressions, findings, conclusion, and recommendation sections; if none of these sections appear in the report, the final report section is used.

[0033] Furthermore, the radiological report uses ScispaCy to tokenize the text before it is provided to the BERT tokenizer.

[0034] Furthermore, the construction of the joint model architecture further includes:

[0035] After processing the input data, a joint model architecture is built. During training, the model inputs predict the severity of edema from images and text using their respective encoders and classifiers, and compares the prediction results with the labels; joint embedding loss is used. Associate the image embedding I with the text embedding R in the joint embedding space.

[0036] Furthermore, in the joint model, the image stream and text stream are decoupled during inference, and only the image stream is used;

[0037] Given a new chest X-ray, an image encoder and classifier calculate the severity of edema.

[0038] Furthermore, the construction of the joint model architecture also includes:

[0039] Train two classifiers and classify severity labels based on image features and text features, respectively.

[0040] As can be seen from the above description, this setup allows us to decouple image classification and text classification during inference, and jointly learning these two representations during training can improve the performance of the image model.

[0041] Furthermore, before the network construction and classification adopt the stochastic gradient-based preprocessing, a preprocessing procedure is set up. The preprocessing procedure includes an image encoder implemented as a series of residual blocks and a text encoder using the BERT model. The image encoder is trained through random initialization, and the BERT model is fine-tuned during joint model training. The BERT model parameters are initialized on the text using pre-trained weights. Image features and text features are represented as 768-dimensional vectors in the joint embedding space, and both classifiers use fully connected layers of 768 to 4.

[0042] Furthermore, the loss in Equation 2 is minimized using the AdamW algorithm based on stochastic gradients, and an adaptive learning rate scheduler is used for learning.

[0043] As described above, this invention first constructs image-text pairs of all data, then builds a joint model architecture, and then applies a ranking-based standard to train the image encoder and the parameterized text encoder to learn the features of the images and text. Subsequently, two fully connected layers (with the same neural network structure) are used in the joint embedding space to assess the severity of edema from the images and reports, respectively. This severity serves as a reference for physicians, thereby further improving the efficiency of physicians in diagnosing patients with acute congestive heart failure.

[0044] Example 1

[0045] A method for auxiliary diagnosis of heart failure based on multimodal data fusion, including...

[0046] Step 1: Preprocessing

[0047] Chest X-rays vary in size, approximately 3000×3000 pixels. During training, images need to be randomly flipped and rotated, and then cropped to 2048×2048 pixels as part of data augmentation. Additionally, the original image resolution is maintained to capture subtle differences between images of varying degrees of pulmonary edema severity. For radiology reports, impressions, findings, conclusion, and recommendation sections need to be extracted. If none of these sections appear in the report, the final report section can be used. Before feeding the text to the BERT tokenizer, we use ScispaCy to tokenize the text.

[0048] Step 2: Building the Joint Model Architecture

[0049] After processing the input data, a joint model architecture is built. During training, the model inputs predict the severity of edema from images and text using their respective encoders and classifiers, and compares the predictions with the labels. Joint embedding loss is used. Image embeddings I are associated with text embeddings R in the joint embedding space. During inference, the image stream and text stream are decoupled, and only the image stream is used. Given a new chest X-ray (image), the image encoder and classifier calculate the severity of its edema.

[0050] Two classifiers are trained, classifying severity labels based on image features and text features, respectively. This setup allows us to decouple image and text classification during inference, and jointly learning these two representations during training improves the performance of the image model.

[0051] Step 3: Learning Joint Expressions

[0052] Applying ranking-based criteria to respectively and A parameterized image encoder is trained to learn image and text feature representations. and Specifically, given an image-text pair Randomly select a replacement image from X. A substitute report (“Replacement” operation). This selection is generated at the beginning of each training phase. Map s(j) produces a random permutation of {1,2,...,N}.

[0053] The goal is to match pairs (I) in the joint embedding space. j ,R j The feature representation between (I) pairs is greater than that between mismatched pairs (I) pairs. s(j) ,R j ) and (I j ,R s(j) The feature representations between I and R are "closer". Directly minimizing the distance between I and R may ultimately push image and text features into a small cluster in the embedding space. Furthermore, the image-text features to be matched are close, while expanding all feature representations for joint modeling of chest X-rays and radiology reports.

[0054] Step 4: Network Construction and Classification

[0055] The image encoder is implemented as a series of residual blocks, while the text encoder is a BERT model. The image encoder is trained with random initialization, while the BERT model is fine-tuned during joint model training. The BERT model parameters are initialized on the text using pre-trained weights. Image and text features are represented as 768-dimensional vectors in the joint embedding space, and both classifiers are 768-dimensional to 4-dimensional fully connected layers.

[0056] The AdamW algorithm, based on stochastic gradients, was then used to minimize the loss in Equation 2, and an adaptive learning rate scheduler was employed for training. All image-text pairs were iterated 10 times by optimizing the first term of Equation 2, and then the labeled image-text pairs were trained for 50 iterations using Equation 2. The mini-batch size was 4. The dot product was used as the similarity metric in Equation 1. The dataset was divided into training and test sets. All hyperparameters were selected based on the results of 5x cross-validation on the training set.

[0057] Formula 1 is:

[0058] Formula 2 is:

[0059]

[0060] After determining the severity of edema, the condition of patients with acute congestive heart failure is linearly reflected.

[0061] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An auxiliary judgment method for heart failure based on multi-modal data fusion, characterized in that, include Preprocessing: Preprocess the chest X-ray and provide the text of the radiology report to the tokenizer; The joint model architecture is built, and the joint embedding loss is calculated after the joint model architecture is built The image embedding I is associated with the text embedding R in the joint embedding space; Joint representation learning, apply ranking-based criteria to and train a parametric image encoder, learn image and text feature representations and given an image-text pair randomly select a substitute image from X and a substitute report Map s(j) produces a random permutation of {1,2,...,N}; match pairs (I j ,R j ) have closer feature representations than mismatch pairs (I s(j) ,R j ) and (I j ,R s(j) ) The network construction and classification adopt the loss based on stochastic gradient minimization in Equation 2 and learns using an adaptive learning rate scheduler; In Formula 1, the dot product is used as its similarity measure; The dataset was divided into training and test sets, and all hyperparameters were selected based on the results of 5x cross-validation in the training set. Formula 1 is: Formula 2 is: After determining the severity of edema, the condition of patients with acute congestive heart failure is linearly reflected.

2. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The chest X-ray image is randomly flipped and rotated during training and cropped to 2048×2048 pixels.

3. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The chest X-ray maintains its original image resolution during preprocessing.

4. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The radiology report extracts the impressions, findings, conclusion, and recommendation sections; if none of these sections appear in the report, the final report section is used.

5. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The radiology report uses ScispaCy to tokenize the text before it is provided to the BERT tokenizer.

6. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The construction of the joint model architecture further includes: After processing the input data, a joint model architecture is built. During training, the model inputs predict the severity of edema from images and text using their respective encoders and classifiers, and compares the prediction results with the labels; joint embedding loss is used. Associate the image embedding I with the text embedding R in the joint embedding space.

7. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 6, characterized in that, During inference, the joint model decouples the image stream and the text stream, and uses only the image stream; Given a new chest X-ray, an image encoder and classifier calculate the severity of edema.

8. The auxiliary diagnostic method for heart failure based on multimodal data fusion according to claim 1, characterized in that, The construction of the joint model architecture further includes: Train two classifiers and classify severity labels based on image features and text features, respectively.

9. The auxiliary diagnosis method for heart failure based on multimodal data fusion according to claim 8, characterized in that, Before the network construction and classification, a pre-process is set up based on stochastic gradient. The pre-process includes an image encoder implemented as a series of residual blocks and a text encoder using the BERT model. The image encoder is trained through random initialization, and the BERT model is fine-tuned during joint model training. The BERT model parameters are initialized on the text using pre-trained weights. Image features and text features are represented as 768-dimensional vectors in the joint embedding space, and both classifiers use fully connected layers of 768 to 4.

10. The auxiliary diagnosis method for heart failure based on multimodal data fusion according to claim 8, characterized in that, The loss in Equation 2 is minimized using the AdamW algorithm based on stochastic gradients, and an adaptive learning rate scheduler is used for learning.

Images