Disease identification method and device based on multi-modal fusion

Abstract

The invention provides a disease identification method and device based on multi-modal fusion, and the method comprises the steps: obtaining structured data, carrying out the feature extraction of thestructured data, and obtaining the features of the structured data; obtaining a case text, and learning the case text by using a weight-shared text embedding module to obtain text features; carryingout bidirectional feature embedding on the structured data features and the text features by using a feature embedding module, carrying out bidirectional attention selection and splicing fusion on thestructured data features and the text features by using a feature fusion module to obtain fused output; acquiring a medical image, performing feature extraction on the medical image, and performing feature space conversion to obtain converted output; fusing the structured features, the text features and the medical image features to obtain an image diagnosis report; and performing full-connectionfeature fusion on the structured features, the fused output, the converted output and the image diagnosis report to obtain a disease identification result.

Description

technical field [0001] The present invention relates to the field of computers, in particular to a disease identification method and device based on multimodal fusion. Background technique [0002] With the improvement of hospital informatization, in order to standardize the writing of medical records and facilitate statistics and research, a large number of hospitals have carried out structured work on medical records, converting the text information of medical records into a series of numerical codes in a fixed format, which reduces the data analysis. And the difficulty of processing provides convenience for predicting respiratory diseases through the analysis of structured medical records by artificial intelligence. The source of respiratory disease diagnosis data based on structured medical records is relatively limited, and the test data and disease description are relatively simple, resulting in low dimensionality of the data. Regression models, support vector machines...

AI Technical Summary

Problems solved by technology

However, only relying on imaging technology will ignore the ability to express the patient's symptoms, resulting in the inability to observe the patient from a macro perspective.

Images