A medical report generation method and system based on multi-modal information fusion

Abstract

The application discloses a medical report generation method and system based on multi-modal information fusion, which extracts features from the multi-modal data of the patient, fuses the features, and further encodes the disease state information, combines the disease state and disease name related information, generates rich disease embedding, inputs the disease embedding into a Transformer decoder with multi-head self-attention and position encoding to generate a medical report, builds a classification network based on a text encoder, summarizes the generated medical report to obtain an abstract embedding, and classifies the abstract embedding into a corresponding disease state; and guides the generator through loss optimization, and the loss of the generator participates in total loss calculation to ensure the accuracy of the report. The method can accurately describe the symptoms, avoid mistakes and omissions, give reliable diagnostic conclusions, and is highly consistent with clinical facts, and can meet the clinical diagnosis needs.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent medical report generation, and relates to a medical report generation method and system based on multi-modal information fusion. BACKGROUND

[0002] The research and development of intelligent medical report generation technology stems from the urgent need for efficiency improvement and standardization of diagnosis in modern medical systems. With the exponential growth of medical image data and electronic medical record scale, and the breakthrough progress of deep learning technology in the field of computer vision and natural language processing, automatic report generation systems based on multi-modal data fusion have gradually become a research hotspot. This technology aims to analyze the visual features of medical images, integrate the text descriptions of electronic medical records, and correlate the time series changes of physiological signals, etc. multi-modal information, to generate structured diagnostic reports that meet clinical standards, thereby alleviating the reality problems of uneven distribution of medical resources and low efficiency of manual report writing.

[0003] However, the existing technical system still faces multiple core challenges: first, traditional medical report generation methods focus on generating readable reports, but the embedding depth of the generation model for the medical knowledge system is insufficient, and the generated text often does not match the clinical facts, with insufficient clinical accuracy and limited credibility; second, previous methods may not have fully utilized multi-modal image information and clinical document information; third, traditional generation models may not have fully considered the impact of disease status on report generation. Specifically:

[0004] 1) Traditional medical report generation methods focus on generating readable reports in terms of design philosophy, expecting to present patient information in clear and fluent text. However, this method has serious defects. The generation model's understanding of the medical knowledge system is superficial, with a serious lack of embedding depth. The medical knowledge system is vast and complex, covering disease pathogenesis, symptom manifestation, diagnostic criteria, and treatment methods, etc. professional content. However, traditional generation models can only learn simple language patterns and text structures, and cannot deeply grasp the internal logic of medical knowledge. This leads to the generated text often not matching clinical facts, such as errors and omissions in describing disease symptoms, or deviations in diagnostic conclusions. This lack of clinical accuracy makes it difficult for doctors to make accurate diagnoses and treatment decisions based on these reports, greatly limiting the credibility and practical application value of the reports.

[0005] 2) With the advancement of medical technology, medical data presents obvious multi-modal characteristics. Multi-modal image information (such as chest X-ray, CT, MRI, etc.) and clinical document information (including medical records, test reports, etc.) contain rich and comprehensive patient health information. However, previous medical report generation methods have failed to fully exploit and utilize this information. When dealing with multi-modal images, only simple preliminary processing and basic feature extraction are performed, which cannot identify subtle lesion areas or early pathological features in the images, resulting in the omission of key diagnostic information. For clinical document information, only keywords or fragments are extracted, lacking deep integration and analysis of patient history, symptom development, and other information. This insufficient utilization of multi-modal information makes the generated medical report unable to fully and accurately reflect the patient's condition, making it difficult to meet the actual needs of clinical diagnosis.

[0006] 3) Disease status plays a key role in medical diagnosis and report generation. Different disease states (such as the progression stage of the disease, the stability or deterioration of the condition, etc.) have a significant impact on the patient's symptoms, treatment plan, and prognosis. However, traditional generation models often ignore the diversity and complexity of disease states when generating medical reports. They usually adopt a single, universal approach to describe diseases, failing to fully reflect the unique characteristics of diseases at different stages. For example, for chronic diseases, early symptoms may be subtle, while late-stage complications may be severe, but traditional models cannot accurately distinguish and describe these differences. This not only makes it difficult for doctors to accurately determine the condition, but also makes the development of treatment plans lack of pertinence, unable to adapt to the treatment needs of patients in different disease states, thereby reducing the clinical practicality and value of medical reports.

[0007] To address the above problems, a technical solution is needed to optimize the performance of medical intelligent report generation and improve its explainability in the field of multi-modal assisted diagnosis and treatment, in order to better meet the application needs of the medical field. Based on this, the present application proposes a medical report generation method and system based on multi-modal information fusion. This method uses techniques such as feature learning, cross-modal alignment, and multi-module collaboration to achieve comprehensive extraction and utilization of multi-modal information, calibration of report content and related disease topics, and generation of reports with both fluency and explainability, further optimizing existing problems in medical report intelligent generation. SUMMARY

[0008] The purpose of the present application is to solve the deficiencies in the generation of medical reports in existing assisted diagnosis and treatment systems, and to provide a medical report generation method and system based on multi-modal information fusion, which can effectively optimize the content, performance, and explainability of generated reports.

[0009] To achieve the above-mentioned purposes of the present application, the present application specifically adopts the following technical solutions:

[0010] A medical report generation method based on multi-modal information fusion, comprising the following steps:

[0011] Step s1: multi-modal data acquisition and preliminary arrangement

[0012] Collecting multi-modal data of patients, and performing preliminary arrangement to ensure the integrity of the data, wherein the multi-modal data comprises one or more of medical record text, medical image, physiological signal and voice complaint;

[0013] Step s2: multi-modal data standardization and preprocessing

[0014] Performing unified standardization operation on the multi-modal data: for medical record text data, performing format unification and character encoding specification to form text data; for medical image data, unifying image resolution and pixel spacing parameters to form image data; for physiological signal data, standardizing sampling frequency and data format to form numerical data; for voice complaint data, performing audio format unification; in addition, performing noise reduction, segmentation and enhancement on the medical image data, and performing noise reduction processing and converting the voice into text form on the voice complaint data, and performing preliminary cleaning and standardization on the text to form text data; finally, forming three types of image data, text data and numerical data;

[0015] Step s3: feature extraction

[0016] Using a machine learning algorithm to perform abnormal diagnosis on the numerical data, and summarizing the abnormal conditions into text data; first extracting visual latent features from the image data through a multi-view image encoder, and processing the text data using a text encoder to extract hidden states to capture semantic information and context relationships in the text;

[0017] Step s4: information fusion and further encoding

[0018] Converting the visual latent features obtained from the image data into visual embeddings, converting the hidden states obtained by processing the text data into text summary embeddings, and then fusing the visual embeddings and the text summary embeddings to form a context disease representation;

[0019] On this basis, further encoding the disease state information, combining the disease state and disease name related information to generate a rich disease embedding, providing a more comprehensive and accurate information basis for subsequent medical report generation;

[0020] Step s5: text generation

[0021] Using the generated rich disease embeddings as initial input, a Transformer decoder with multi-head self-attention and positional encoding is employed. During the generation process, the probability of the next word is predicted based on the preceding word and the rich disease embeddings in the input. By fusing the word embeddings from the previous step with the disease embeddings, and through multiple layers of calculation and filtering, the text content of the medical report is gradually generated.

[0022] Step s6: Explanation, evaluation, and feedback optimization

[0023] A classification network is built based on a text encoder to summarize the generated medical reports, obtain summary embeddings, and classify them into the corresponding disease states; and a loss optimization is used to guide the generator, whose loss is used in the total loss calculation to ensure the accuracy of the reports.

[0024] In the above technical solution, further, in step s3, for each item containing Using data from multiple images, a multi-view image encoder is used to extract latent features from each image. By performing max pooling on these latent features, multi-view latent features are obtained. ;

[0025] For the text encoder, the input text data is transformed into a format of length [length missing] using the Word2Vec model. Word embedding The data is then further input into a text feature extractor composed of a Transformer encoder to retrieve a set of hidden states. ,in Representing the The attention features of each word relative to other words in the text, that is, the degree of correlation between each word vector.

[0026] Furthermore, in step s4, the latent visual features obtained from the image encoder are... Decoupling transforms the disease into a low-dimensional representation, serving as a visual embedding. The specific process involves embedding visual information. Each row is represented as a vector , and These are the learnable parameters representing the j-th disease, obtained through parameter tuning during model training;

[0027] pass Get text summary embedding Where H is the hidden state obtained from the text encoder, and Q is the set of vectors. The term "formation" represents the n disease-related topics to be queried from the input text data. Their initial values ​​are randomly set and are subsequently learned during the attention mechanism's operation. is the word attention heat map for n query diseases in the document;

[0028] The visual embedding and the text summary embedding are fused to form the contextual disease representation , which is formulated as .

[0029] Further, the rich disease embedding is generated by

[0030] Let k be the number of states, and S be the state embedding. The confidence of classifying each disease into one of the k disease states is calculated by , which is used as a feature for multi-label classification to calculate the classification loss , where and are the jth real value and predicted value of the ith disease, respectively. The state-aware embedding is calculated according to the training and testing phases, respectively: training phase , testing phase , is the real label of the disease-related topic, is the predicted value.

[0031] The rich disease embedding is composed of the state-aware disease embedding , the disease name , and the disease representation , which is formulated as , where the disease name is randomly initialized and learned during training.

[0032] Further, in step s5, the report generation is based on a Transformer encoder, which is composed of a secondary stacked mask multi-head self-attention component and a feedforward layer.

[0033] Based on the hidden state H obtained in the text encoder, the model predicts the next word, and the prediction process is implemented by the formula , where is the entire vocabulary embedding, v is the size of the vocabulary, and e is the embedding dimension. This vocabulary embedding is a large word library already existing in the dataset, containing all the words needed for report generation. The Softmax function converts the hidden state into a probability distribution, so that each word in the vocabulary has a corresponding probability, and the word with the highest probability is considered to be the next word predicted by the model.

[0034] To measure the difference between the generated word and the real word, the generator loss is defined as the difference between the generated word and the real word and prediction words The cross-entropy is ,in, This indicates whether the real word at position i is the j-th word in the vocabulary; 1 indicates yes, 0 indicates no. It is the probability that the model predicts the j-th word in the vocabulary at position i; this is achieved by minimizing the generator loss. Adjusting the generator parameters to make the generated medical reports more closely resemble real medical reports; finally, weighted word embedding. The generated report, through The calculation shows that this process combines the probability distribution of predicted words with vocabulary embeddings to obtain a word embedding representation of the generated medical report, which can be used for subsequent processing or evaluation.

[0035] Furthermore, in step s6, an interpreter is constructed, which is a classification network built based on the text encoder, to process the generated medical report and analyze the degree of matching between the report content and disease-related topics.

[0036] The text encoder summarizes the currently generated medical report. Output a report summary of the disease Q that was queried. , It is based on From the generated medical report The calculated hidden state represents the result of the encoder processing the current word. and the words before ~ and disease embedded information ~ Encode; This is a word attention heatmap for n queried diseases in the report. The principle is to calculate the vector similarity dot product between the queried disease and the words in the report to obtain the degree of association between each disease and the words in the report. Then, these words are weighted and summed to obtain a summary of the report features for the queried diseases. ;

[0037] Embed a summary of each report That is, matrix Each row is classified as one of k disease-related states. Calculate Where S is the state embedding, This represents the probability distribution of classifying each disease-related topic into different states.

[0038] Furthermore, the interpreter loss is calculated and the interpreter is trained: the goal of training the interpreter is to minimize the multi-label classification loss. ,in, These are real disease labels, representing the j-th real state of the i-th disease. is the predicted disease label of the interpreter, i.e., the probability that the i-th disease is predicted to be the j-th state; by minimizing this loss, the parameters of the interpreter are adjusted so that it can more accurately judge the disease state according to the generated report;

[0039] In fine-tuning the generated medical report , all parameters of the interpreter are frozen, at this time, the interpreter is equivalent to a fixed evaluation standard, and the guide words represent the representation learned by the interpreter from real medical reports.

[0040] A medical report generation system based on multi-modal information fusion, comprising:

[0041] A classification module: used for feature extraction of processed multi-modal data respectively, fusion, formation of context disease representation, and further encoding of disease state information, combination of disease state, disease name related information, and generation of rich disease embedding;

[0042] A generation module: taking the generated rich disease embedding as the initial input, using a Transformer decoder with multi-head self-attention and position encoding, in the generation process, predicting the probability of the next word according to the previous word in the rich disease embedding and the input embedding, through the fusion of the word embedding and the disease embedding in the last step, and through multi-layer calculation and screening, gradually generating the text content of the medical report;

[0043] An explanation module: used for checking and fine-tuning the text generated by the generation module, comparing and analyzing the text generated by the generation module with the disease-related topics generated by the classification module, checking whether the text content is consistent with these topics; if it is found that there are places in the generated text that do not match the disease-related topics, the text is fine-tuned.

[0044] An electronic device, comprising:

[0045] One or more processors;

[0046] Memory for storing one or more programs;

[0047] When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of the above.

[0048] A computer-readable storage medium storing computer-executable instructions, the instructions being executed to implement the method of any one of the above.

[0049] The present application has the following beneficial effects relative to the prior art:

[0050] (1) In the traditional medical report generation method, the generation model has a shallow embedding of the medical knowledge system, only learns simple language patterns and text structures, and is difficult to grasp the internal logic of medical knowledge, resulting in generated text often inconsistent with clinical facts, poor clinical accuracy. The invention uses advanced deep learning architectures such as Transformer and its multi-head self-attention mechanism, position encoding, etc., combined with a large amount of medical professional data to deeply train the model, allowing the model to deeply understand medical knowledge. When generating medical reports, it can accurately describe symptoms, avoid mistakes, and give reliable diagnostic conclusions, which are highly consistent with clinical facts, have report credibility and practical application value, and solve the problem of insufficient accuracy of clinical automatic report generation.

[0051] (2) Previous medical report generation methods do not fully utilize multi-modal images (such as chest X-rays, CT, MRI, etc.) and clinical document information, and images are only preliminarily processed, missing key lesion information, and clinical documents only extract information on the surface, unable to fully reflect the patient's condition. The invention analyzes and extracts feature information from multi-modal image data, clinical document text data, and physiological signal numerical data, and uses a specific fusion algorithm to organically integrate multi-modal information. The generated medical report can fully and accurately present the patient's condition, providing doctors with rich and accurate information, improving medical diagnosis quality and efficiency, and meeting the needs of clinical diagnosis.

[0052] (3) Traditional generation models ignore the diversity of disease states and use a single approach to describe diseases, making it difficult for doctors to accurately determine the condition and lack of targeted treatment plans. The invention builds a disease state model based on deep learning, uses a large amount of medical data labeled with disease state information to train, and integrates the output of the disease state model and other features in the generation model, using Transformer decoder and attention mechanism, etc. to accurately and comprehensively describe symptoms, develop reasonable treatment plans, and scientifically predict prognosis according to different disease states. For example, for chronic diseases, clearly distinguish between early and late stage characteristics and describe in detail, helping doctors accurately determine the condition and develop personalized treatment plans, adapting to the treatment needs of patients with different disease states, improving the clinical practicality and value of medical reports, and providing high-quality medical services for patients.

[0053] (4) The medical report generated in the traditional method may not match the actual condition, and there is a lack of effective calibration mechanism. The explanation module of the present application relies on the text encoder based on Transformer to build a classification network, adopts a multi-label cross-entropy loss function to measure the difference between the classification result and the true disease state, fixes part of the parameters during fine-tuning (basic algorithms such as gradient descent can be used), and adjusts the parameters according to the loss value to make the generated report word vector close to the true report, and the loss of the explanation module is included in the total loss calculation of the end-to-end training of the model. The module effectively calibrates the medical report, makes the report semantics and content consistent with the disease theme, improves the reliability and clinical accuracy of the report, and provides a reliable reference for medical diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 It is a flowchart of the method and system of the present application.

[0055] Figure 2 It is a schematic diagram of the multi-module cooperative generation in a specific example of the method of the present application.

[0056] Figure 3 It is a structural block diagram of the medical report intelligent generation system of the present application.

[0057] Figure 4 It is a comparison chart of the report after the explanation module, the true situation and the original report in one generation example in the method of the present application. DETAILED DESCRIPTION

[0058] The technical solutions of the present application will be further described in detail below in combination with the drawings.

[0059] The present application provides a medical report intelligent generation method, which is realized based on multi-modal information fusion, and correspondingly provides a medical report intelligent generation system, the core of which includes a classification module, a generation module and an explanation module. The specific process of the method is mainly as shown in Figure 1 . Figure 1 It is a flowchart of the first embodiment of the medical report intelligent generation method of the present application.

[0060] The following is the detailed process of one embodiment of the method, which covers four parts of multi-modal data preprocessing, classification module, generation module and explanation module, and the principles of each module are described:

[0061] Multi-modal data preprocessing

[0062] 1. Data Collection: Collecting the patient's electronic medical record text, which contains detailed records of the patient's basic information, medical history, symptom description, test results, etc.; medical images such as chest X-rays, CT, MRI, etc., which can present the internal structure of the patient's body and potential lesions; physiological signals such as electrocardiogram (reflecting heart electrical activity), blood oxygen data (reflecting oxygen content in blood); voice complaints, i.e. patients describe their discomfort and symptoms through voice.

[0063] 2. Standardization Processing: For electronic medical record text, format uniformity is performed, such as standardizing paragraph format, punctuation usage, etc., while unifying character encoding to ensure correct reading and processing between different systems and devices. For medical image data, parameters such as image resolution and pixel spacing are unified to make images obtained by different devices comparable. Numerical data (such as physiological signal data) is standardized in terms of sampling frequency and data format to ensure consistency and accuracy. For voice complaint data, it is converted to a unified audio format and further converted to text form through speech recognition technology. Machine learning algorithms are used to process numerical data and identify possible abnormalities, which are summarized as text data.

[0064] 3. Unstructured Data Processing: For medical images, noise reduction algorithms (such as median filter, Gaussian filter, etc.) are used to remove noise interference and improve image quality; image segmentation techniques (such as threshold segmentation, region growing, etc.) are used to extract regions of interest such as lungs, hearts, and other specific organs for subsequent analysis; image enhancement algorithms (such as histogram equalization, etc.) are used to highlight image details and make lesions more obvious. For text data converted from voice, cleaning and standardization are performed to remove irrelevant characters and errors.

[0065] Classification Module

[0066] 1. Image Feature Extraction: Use multi-view image encoder, for example, DenseNet-121, to extract latent features from image data. DenseNet-121 uses dense connections to better pass and reuse feature information, thus extracting rich details from images. Perform max-pooling operation on the extracted features to reduce feature dimension while preserving important feature information, obtaining multi-view latent features.

[0067] 2. Text Feature Extraction: Use text encoder (such as Transformer-based encoder) to process text data. Transformer encoder uses self-attention mechanism to capture semantic associations between different positions in the text, thus obtaining hidden states of clinical documents and extracting key semantic information.

[0068] 3. Information Fusion and Further Encoding: The visual latent features are transformed into visual embeddings, and the text hidden states are transformed into text summary embeddings. These two are fused by addition, concatenation, etc., to form a contextual disease representation, making the image and text information complementary. On this basis, further encode the disease state information, combine the disease state, disease name, and other related information to generate a rich disease embedding. This process uses attention mechanisms and other technologies to highlight important information related to the disease, providing a more comprehensive and accurate basis for subsequent medical report generation.

[0069] Generation Module

[0070] 1. Initial Input and Architecture: The rich disease embedding generated by the classification module is used as the initial input, and a Transformer decoder with multi-head self-attention and position encoding is used. Multi-head self-attention mechanism allows the model to capture the dependencies between elements in the input sequence in different representation subspaces, better understanding the context information; position encoding provides the model with the position information of elements in the sequence, as the Transformer itself does not have the ability to perceive the sequence order.

[0071] 2. Word Prediction and Generation Process: When generating a medical report, according to the previous word and input embedding (combined with disease embedding and position encoding), the attention weights between the current position and other positions are calculated through multi-head self-attention mechanism to determine which information is more important for predicting the next word. Then, through a feedforward neural network, the scores of each word in the vocabulary are calculated, and the probability distribution of the next word is obtained through the softmax function. The word with the highest probability (or using sampling methods) is selected as the next word generated, and this process is repeated to gradually generate the text content of the medical report. During the generation process, word embeddings and disease embeddings are also fused, and after multiple layers of calculation and screening, the generated text conforms to medical knowledge and language logic.

[0072] Explanation Module

[0073] 1. Classification Network Building and Feature Extraction: A classification network is built based on the text encoder, and the generated medical report is input into this network. The text encoder further processes the medical report to extract the summary embedding, which contains the core semantic information of the medical report.

[0074] 2. Disease State Classification and Loss Calculation: The summary embedding is input into the classification layer of the classification network, and according to the pre-defined disease state categories, the probability distribution of the summary embedding belonging to each disease state is calculated. A multi-label classification loss function (such as cross-entropy loss function) is used to measure the difference between the classification result and the true disease state. The true disease state is determined through clinical diagnosis, etc., and is represented in the form of a label vector.

[0075] 3. Fine-tuning and guidance generation: In the fine-tuning stage, part of the parameters of the interpreter (classification network) are fixed, and the loss value calculated is used to guide the generator. The loss value reflects the deviation between the generated medical report and the true situation. Through the back propagation algorithm, the loss value is transmitted to the generator, and the parameters of the generator are adjusted, so that the generated report word vector gradually approaches the word vector of the true report. The loss of the interpretation module is involved in the total loss calculation of the entire model end-to-end training. By continuously adjusting the model parameters, it ensures that the generated medical report is consistent with the disease theme in terms of semantics and content, and improves the clinical accuracy and reliability of the medical report.

[0076] Embodiment 1

[0077] A medical report generation method based on multi-modal information fusion, as shown in Figure 2 , comprising the following steps:

[0078] Step 1: Multi-modal data acquisition and preliminary arrangement

[0079] Collect the patient's electronic medical record text, medical images (X-ray / CT / MRI), physiological signals (electrocardiogram, blood oxygen), voice complaints and other multi-modal data, and perform preliminary arrangement and record to ensure data integrity and traceability. Classify and identify the data preliminarily for subsequent processing.

[0080] Step 2: Multi-modal data standardization processing

[0081] Perform uniform standardization operation on the collected heterogeneous multi-modal data. For electronic medical record text, perform format unification, character encoding specification, etc.; for medical image data, unify image resolution, pixel spacing, etc. Parameters; for physiological signal data, standardize sampling frequency, data format, etc.; for voice complaint data, perform audio format unification, etc.

[0082] Step 3: Preprocessing of unstructured data

[0083] Perform noise reduction, segmentation, enhancement and other processing on unstructured data (such as voice, image). For medical images (X-ray CT / MRI), use appropriate algorithms to remove noise and improve image quality; through image segmentation technology, extract the region of interest (such as the lesion area); use image enhancement algorithm to highlight the detailed features of the image. For voice complaint data, perform noise reduction to remove environmental noise interference, then convert the voice to text form, and perform preliminary cleaning and standardization on the text.

[0084] Step 4: Feature extraction of classification module

[0085] First, the multi-view image encoder is used to extract the latent features from the chest X-ray images (in this example, the image data is only taken as an example of X-ray) using image encoders such as DenseNet, etc., to reduce the feature dimension and retain important information. At the same time, the clinical documents are processed using text encoders such as LSTM, Transformer encoder, etc., to capture the semantic information and contextual relationships in the text. Using neural networks such as FCN or CNN, and using machine learning algorithms such as random forests, the patient's laboratory data and other digital modalities of physiological signals are processed and possible abnormalities are obtained, summarized as text data, which are also processed using text encoders to capture semantic information and contextual relationships in the text. Finally, all modal data are combined for training and generation.

[0086] The multi-view image encoder is a common image encoding processing method in the medical field, and its purpose is to encode multiple images representing the same part or disease together. The specific process is as follows: for each item containing chest X-ray images, a shared DenseNet-121 image encoder is used to extract the latent features of each image . Then, by performing maximum pooling operation on these latent features, the multi-view latent features are obtained. When , the multi-view encoder is simplified to a single image encoder.

[0087] For the text encoder, the input text document is converted into a word embedding with a length of by the Word2Vec model, and further input into the text feature extractor composed of the Transformer encoder to retrieve a set of "hidden states" , where represents the attention features of the th word relative to other words in the text. That is, it represents the degree of association between each word vector.

[0088] Step 5: Information fusion and further encoding of the classification module

[0089] The visual latent features obtained from the chest X-ray images are converted into visual embeddings, and the hidden states obtained by processing the text data are converted into text summary embeddings. Then the visual embeddings and text summary embeddings are fused to form a contextual disease representation, so that the information of different modalities can complement each other. On this basis, the disease state information is further encoded, combining disease status, disease name and other related information to generate rich disease embeddings, providing a more comprehensive and accurate information basis for subsequent medical report generation.

[0090] where the latent visual features obtained in the image encoder are decoupled into low-dimensional disease representations as visual embeddings . The detailed procedure is to represent each row of the visual embeddings as a vector , and are learnable parameters of the jth disease representation, which are obtained by tuning in the training of the model.

[0091] The text summary embeddings are obtained by . Here, H is the "hidden state" obtained in the previous text encoder, and Q is formed by a set of vectors representing n disease-related topics to be queried from the input document, whose initial values are randomly set and learned in the subsequent running of the attention mechanism. Here, is the word attention heat map for n query diseases in the document, which is based on the principle that for each disease (e.g., pneumonia) queried from the text file T, the model only focuses on the most critical words (e.g., cough, shortness of breath) related to the disease in the text, and uses vector similarity dot product to calculate the degree of association between these words and the disease. In this way, by weighting and summing these words according to the formula, the document feature summary for the query disease can be obtained, and these summary information corresponds to the n disease-related topics, which constitutes Q.

[0092] Then, the visual embeddings and the text summary embeddings are fused by "residual connection and layer normalization" operation to form the contextual disease representations . The formula is . This fusion method can combine visual and textual information, simulate the hospital workflow, and improve the performance of the medical report generation task.

[0093] Generating rich disease embeddings: To further encode disease state information such as positive, negative, uncertain, or not mentioned, let k be the number of states, and S be the state embedding. The confidence of classifying each disease into one of the k disease states is calculated by the formula , and the classification loss is calculated as a multi-label classification feature , where and are the jth true value and predicted value of the ith disease, respectively; the state-aware embedding is calculated according to the training phase and the test phase, respectively: training phase , test phase , is the true label of the disease-related topic, is the predicted value; finally, the rich disease embedding from the state-aware disease embedding , disease name (randomly initialized and learned during training) and disease representation consists of, formula .

[0094] Step 6: Text generation of the generation module

[0095] The rich disease embedding generated by the classification module is used as the initial input, and a Transformer decoder with multi-head self-attention and position encoding is used. In the generation process, the probability of predicting the next word is calculated based on the previous word and the input embedding. By fusing the word embedding and disease embedding of the previous step, the medical report text content is gradually generated through multiple layers of calculation and screening.

[0096] The report generator of the generation module is based on the Transformer encoder. The Transformer encoder consists of a sub-stack of masked multi-head self-attention components and feed-forward layers. This architecture can effectively process sequence data and capture dependencies between different positions in the input sequence, which is crucial for generating coherent and accurate medical reports.

[0097] Word prediction: based on the hidden state H calculated earlier, the model predicts the future word. The prediction process is implemented by the formula , where is the entire vocabulary embedding, v is the size of the vocabulary, and e is the embedding dimension. This vocabulary embedding is a large word library already available in the dataset, containing all the words needed for report generation. The Softmax function converts the hidden state into a probability distribution, so that each word in the vocabulary has a corresponding probability. The word with the highest probability is considered to be the model's prediction of the next word.

[0098] Generator loss calculation: to measure the difference between the generated word and the true word, the generator loss is defined as the cross-entropy between the true word and the predicted word , formula . Where, indicates whether the true word at position i is the jth word in the vocabulary (1 for yes, 0 for no), is the probability that the model predicts the jth word in the vocabulary at position i. By minimizing the generator loss , the parameters of the generator can be adjusted so that the generated medical report is closer to the true medical report.

[0099] Weighted word embedding calculation: finally, the weighted word embedding (i.e., the generated report) is computed by the formula . This process combines the predicted probability distribution of the words with the vocabulary embeddings to obtain a word embedding representation of the generated medical report for subsequent processing or evaluation.

[0100] Step 7: Evaluation and feedback optimization of the explanation module

[0101] A classification network is built based on the text encoder to summarize the generated medical report and classify it into the corresponding disease state. During training, the multi-label classification loss is minimized, and the interpreter parameters are fixed during fine-tuning. The loss value is used to guide the generator to produce more accurate word representations, making the generated report word vectors closer to the true report. The loss of this module participates in the total loss calculation of the end-to-end training of the model, and by continuously adjusting the model parameters, it ensures that the generated report is consistent with the disease theme, improving the clinical accuracy and reliability of the medical report.

[0102] The explanation module builds a classification network based on the text encoder to calibrate the medical report and improve clinical accuracy. By inputting the generated report, querying disease information, state information, and true disease conditions, it outputs a summary of the report disease information, disease state probability, examination loss value, and adjusted report. Finally, it outputs a more interpretable and professional report. The principle is as follows:

[0103] Building a classification network foundation: The explanation module builds a classification network based on the text encoder (the same as the text encoder used to process clinical documents in the classification module, using a Transformer encoder). This network is used to process the generated medical report to analyze the matching degree of the report content and the disease-related theme.

[0104] Processing the generated medical report:

[0105] The text encoder summarizes the current generated medical report ( is the weighted word embedding output by the generation module, i.e., the generated report). In this process, the report summary embedding .

[0106] Here, is the hidden state calculated from the generated medical report according to the formula . is the word attention heat map for the n query diseases in the report. The principle is to calculate the vector similarity dot product between the query diseases and the words in the report to obtain the correlation degree between each disease and the words in the report, and then perform weighted summation on these words to obtain the report feature summary for the query disease .

[0107] Disease state classification: Each report summary embedding is classified into one of the k disease-related states, computed as , where S is the state embedding (introduced in the classification module to encode disease state information), represents the probability distribution of classifying each disease-related topic into different states. Computing interpreter loss and training: The goal of training the interpreter is to minimize the multi-label classification loss . Here,

[0108] is the true disease label (denoting the jth true state of the ith disease), is the interpreter's predicted disease label (i.e., the probability of the ith disease being predicted as the jth state). By minimizing this loss, the interpreter's parameters are adjusted so that it can more accurately judge disease states based on the generated reports. Fine-tuning generated medical reports: When fine-tuning the generated medical reports , all parameters of the interpreter are frozen. At this time, the interpreter serves as a fixed evaluation standard, and the guide words represent

[0109] the representations learned by the interpreter from real medical reports. In this way, the generated medical reports are adjusted to be more consistent with the output of the classification module, improving the clinical accuracy of the reports. Embodiment 2

[0110] A medical report generation system based on multi-modal information fusion includes three modules:

[0111] 1. Classifier module: This classifier reads images such as chest X-rays and patient clinical history information. By analyzing chest X-rays and other images, it identifies possible disease-related image features, such as lung shadows, nodules, etc. At the same time, it processes information such as patient symptoms, past medical history, treatment, etc. recorded in the clinical history. Then, based on the comprehensive analysis of these multi-modal information, an internal list of disease-related topics is generated. This list lists possible disease types, potential health problems, and other related topics, providing a basis and direction for subsequent processing.

[0112]

[0113] ​​2. Transformer-based generator module: This generator is based on the Transformer architecture. The Transformer architecture has strong capabilities in processing sequential data, capable of capturing long-distance dependencies and semantic information in text. The generator uses the disease-related topic list generated by the classifier and other information to generate fluent and natural text content. These text contents may include descriptions of patient conditions, preliminary inferences of diagnoses, etc., in an easy-to-understand language form, providing valuable references for doctors or relevant personnel.

[0114] 3. Interpreter module: The role of the interpreter is to check and fine-tune the text generated by the generator. It will compare and analyze the generated text with the disease-related topics generated by the classifier to check if the text content is consistent with these topics. If it is found that the generated text does not match the disease-related topics, for example, there are description errors, important information is missing, or unrelated content appears, the interpreter will fine-tune the text by modifying, supplementing, or deleting, etc. operations, so that the generated text is highly consistent with the disease-related topics in content and semantics, thereby improving the accuracy and reliability of the text.

[0115] These three modules work together, the classifier provides disease-related topic information, the generator generates text based on this information, and the interpreter optimizes the generated text to ensure that the final generated text meets the actual situation and requirements related to the disease. It constitutes the core of the medical report intelligent generation system, namely the reasoning module, in addition, the system also includes input module and memory module, as Figure 3 shown.

[0116] Input module:

[0117] The input module is the starting key link of the multi-modal medical model data processing flow, and is specially used to realize the operation of users inputting data into the multi-modal medical model and triggering the processing program. In view of the wide data sources and various forms in the medical scene, the module provides users with a convenient and comprehensive information uploading channel. Users can upload various modal information with the help of this module. In terms of medical image images, whether it is X-ray film, CT image, MRI image, or ultrasound image, etc., the internal organization form and potential lesions of the patient's body can be clearly presented, providing intuitive visual basis for subsequent diagnosis; medical history covers the detailed records of patients since the first medical treatment, including the initial symptoms of the disease, the development process, the treatment methods once received and the treatment effect, etc. These information is crucial for doctors to understand the evolution of the disease and grasp the law of the disease; laboratory indicators include the detection data of various samples such as blood, urine and body fluid, such as blood sugar, blood lipid, liver function index, kidney function index, etc., which reflect the physiological function and health status of the patient from the micro level. After receiving these different modal data, the module will use advanced data processing technology to accurately input and fuse multi-modal information. Through a series of operations such as data standardization, feature extraction, format conversion, etc., different types of data are converted into a unified and convenient format for model analysis, laying a solid foundation for the efficient operation of multi-modal medical models, and ensuring that the model can fully utilize these information for accurate diagnosis and analysis.

[0118] The reasoning module:

[0119] The reasoning module occupies a core position in the multi-modal medical model system. As mentioned earlier, it is responsible for transmitting various types of information uploaded by users to the multi-modal medical model inside to automatically generate medical reports.

[0120] The memory module:

[0121] The memory module is an important component of a multi-modal medical system for data storage and management. Its main function is to store user-uploaded data and various types of data generated by the model, thereby building a rich and valuable historical database that provides strong support for subsequent diagnostic work. In practical applications, the memory module can accurately and completely save data records at different time nodes by establishing a detailed patient historical medical record library. These records include not only user-uploaded basic data such as original medical images, medical records, and laboratory reports, but also important information such as medical reports generated by the model at different stages. The memory module uses advanced data storage technology and management strategies to ensure data security, stability, and retrievability. At the same time, it provides users with convenient historical data query functions, allowing users to browse and retrieve patient's past medical information at any time, understand the development and change process of the disease, and better grasp the disease dynamics. In addition, the memory module provides flexible data calling interfaces, and the reasoning module can easily obtain and utilize historical data in subsequent diagnostic processes, combining it with current detection information for comprehensive analysis. In this way, the multi-modal medical model can fully consider the patient's disease development history, avoid misdiagnosis or missed diagnosis due to information loss or one-sidedness, ensure the continuity and accuracy of model diagnosis, and provide a more comprehensive and reliable basis for doctors to develop scientific and effective treatment plans, helping to improve the quality and level of medical services.

[0122] Figure 4 A comparison chart of the report after the explanation module in the method of the present application with the true situation and the original report. The chart involves two cases above and below. It can be seen that the report after the explanation module of the present application has the following advantages:

[0123] I. Significant improvement in accuracy

[0124] Correcting the deviation of the original report: The original report has information misjudgment or loss. For example, in the first row of the case, the original report does not mention the key feature of "heart contour enlargement again", which is a true annotation, while the report after the explanation module clearly supplements "heart contour enlargement" and associates "pacing lead extending to the right atrium and right ventricular apex", making the description consistent with the true condition.

[0125] Correcting ambiguous expressions: In the second row of the case, the original report only generally compares the changes in the lung window and does not accurately reflect the true details such as "improvement of bilateral atelectasis" and "right PICC tip located in the superior vena cava". The report after the explanation module accurately restores these key information, avoiding misguidance.

[0126] II. Enhanced content integrity

[0127] Supplement missing information: The original report often misses important details of the disease. For example, in the first case, "no clear blood vessel congestion, but there is a possibility of underlying heart disease or pleural effusion" and "PICC has been removed" are real annotations, which are completely missing in the original report, but the report after the explanation module is complete, covering the whole picture of the disease.

[0128] Integrate multi-dimensional information: The explanation module integrates the multi-dimensional features of the image, such as the second case, which not only states "the lung window volume is low, and there is a small amount of pleural effusion in the left lung", but also supplements "there is no pneumothorax on the right side", so that doctors can fully understand the lung condition, while the original report only focuses on a single dimension (such as pulmonary vascular resistance).

[0129] Three, logical correlation optimization

[0130] Establish symptom association: The report after the explanation module pays more attention to the logical association of medicine. For example, in the first case, "enlarged heart contour" is combined with "pacing lead extended to the right atrium and right ventricular apex", which reflects the association between imaging features and clinical catheter operation; while the original report only describes "enlarged heart" in isolation, without digging into the potential logical connection, making it difficult to support in-depth clinical analysis.

[0131] The above-described embodiments are only some of the preferred schemes of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the scope of protection of the present application.

Claims

1. A method for medical report generation based on multi-modal information fusion, characterized in that, The method comprises the following steps: Step s1: multi-modal data acquisition and preliminary arrangement Collecting multi-modal data of the patient, and performing preliminary arrangement to ensure the completeness of the data, wherein the multi-modal data comprises one or more of medical record text, medical image, physiological signal and voice complaint; Step s2: multi-modal data standardization and preprocessing Performing unified standardization operation on the multi-modal data to form three categories of image data, text data and numerical data; Step s3: feature extraction Using a machine learning algorithm to perform abnormal diagnosis on the numerical data, and summarizing the abnormal conditions into text data; first extracting visual latent features from the image data through a multi-view image encoder, and performing maximum pooling operation, using a text encoder to process the text data, and extracting hidden states to capture semantic information and context relationship in the text; Step s4: information fusion and further encoding The visual potential features obtained from the image data are converted into visual embeddings, the hidden states obtained by processing the text data are converted into text summary embeddings, and then the visual embeddings and the text summary embeddings are fused to form the context disease representation ; On this basis, further encode disease state information, combine disease state, disease name related information to generate rich disease embedding; including: set k as state quantity, S as state embedding, through formula Calculate the confidence of classifying each disease into one of the k disease states, state-aware embedding According to the training phase and the test phase, respectively: training phase , test phase , is the true label of the disease-related topic, is the predicted value; rich disease embedding is composed of state-aware disease embedding , disease name and disease representation , for , wherein the disease name is randomly initialized and learned in training; Step s5: text generation Taking the generated rich disease embedding as initial input, adopting a Transformer decoder with multi-head self-attention and position encoding, in the generation process, predicting the probability of the next word according to the previous word and the input rich disease embedding, through the fusion of the word embedding and the disease embedding of the last step, gradually generating the text content of the medical report through multi-layer calculation and screening; Step s6: explanation evaluation and feedback optimization Relying on the text encoder to build a classification network, summarizing the generated medical report to obtain an abstract embedding, and classifying it into the corresponding disease state; and guiding the generator through loss optimization, and the loss of the generator participates in the total loss calculation to ensure the accuracy of the report. 2.The method of claim 1, wherein, In step s3, for each item of data comprising a plurality of images, latent features of each image are extracted using a multi-view image encoder , and multi-view latent features are obtained by performing a max-pooling operation on the latent features ; For text encoder, the input text data is transformed into word embeddings of length by Word2Vec model and further input into the text feature extractor composed of Transformer encoder to retrieve a set of hidden states , where represents the attention feature of the th word relative to other words in the text, i.e., represents the degree of association between each word vector. 3.The method of claim 2, wherein, In step s4, the latent visual features obtained in the image encoder are decoupled into low-dimensional disease representations as visual embeddings Decoupling translates into low-dimensional disease representations as visual embeddings The specific process is to represent each row of the visual embedding as a vector , and are learnable parameters of the jth disease representation, obtained by tuning in the training of the model; By obtaining a text summary embedding where H is the hidden state obtained in the text encoder, Q is formed by a set of vectors representing n disease-related topics to be queried from the input text data, whose initial values are randomly set and subsequently learned during the running of the attention mechanism, is the word attention heat map in the document for the n query diseases; The visual embeddings are fused by residual connections and layer normalization operations and the text summary embeddings to form a contextual disease representation whose formula is . 4.The method of claim 3, wherein, The rich disease embedding is generated by: Let k be the number of states, S be the state embedding, the confidence of classifying each disease into one of the k disease states is calculated by the formula As a feature of multi-label classification, the classification loss is calculated where and are the jth real value and the prediction value of the ith disease, respectively; the state-aware embedding is the real label of the disease-related topic, is the predicted value;​​​​ rich disease embeddings from state-aware disease embeddings , disease name and disease representation consisting of wherein the disease name is randomly initialized and learned in training. 5.The multi-modal information fusion based medical report generation method of claim 1, wherein, In step s5, the report generation is based on a Transformer encoder, which is composed of a secondary stacked mask multi-head self-attention component and a feedforward layer; Based on the hidden state H obtained in the text encoder, the model predicts the future word, and the prediction process is through the formula is implemented, wherein is the entire vocabulary embedding, v is the vocabulary size, and e is the embedding dimension. This vocabulary embedding is a large word library already existing in the data set, containing all the words needed for report generation; the Softmax function converts the hidden state into a probability distribution, so that each word has a corresponding probability in the vocabulary, and the word with the highest probability is considered to be the next word predicted by the model; To measure the difference between the generated words and the true words, the generator loss is defined as the cross-entropy between the true words and the predicted words , which is where indicates whether the true word at position i is the j-th word in the vocabulary, and is 1 or 0 otherwise, is the probability that the model predicts the j-th word in the vocabulary at position i; by minimizing the generator loss , the parameters of the generator are adjusted so that the generated medical report is closer to the true medical report; finally, the weighted word embedding , i.e., the generated report, is calculated by , which combines the probability distribution of the predicted words with the vocabulary embedding to obtain the word embedding representation of the generated medical report for subsequent processing or evaluation. 6.The method of claim 1, wherein, In step s6, an interpreter is built, which is a classification network based on a text encoder, used to process the generated medical report to analyze the matching degree of the report content and the disease-related theme; The text encoder summarizes the currently generated medical report. Output a report summary of the disease Q that was queried. , It is based on From the generated medical report The calculated hidden state represents the result of the encoder processing the current word. and the previous words ~ and disease embedded information ~ Encode; This is a word attention heatmap for n disease queries in the report. The principle is to calculate the vector similarity dot product between the query disease and the words in the report to obtain the correlation between each disease and the words in the report. Then, these words are weighted and summed to obtain a summary of the report features for the query disease. ; embedding each report summary i.e. matrix each row of which is classified into one of k disease-related states, compute where S is the state embedding, denotes the probability distribution of classifying each disease-related topic into a different state.

7. The multi-modal information fusion based medical report generation method of claim 6, wherein, Compute the interpreter loss and train: The goal of training the interpreter is to minimize the multi-label classification loss where, is the true disease label, indicating the jth true state of the ith disease, is the interpreter's predicted disease label, i.e., the probability that the ith disease is predicted to be in the jth state; by minimizing this loss, the interpreter's parameters are adjusted so that it can more accurately judge the disease state according to the generated report; In fine-tuning the generated medical reports all parameters of the interpreter are frozen, at which point the interpreter is equivalent to a fixed evaluation criterion, the lead words represent the representations learned by the interpreter from real medical reports. 8.A medical report generation system based on multi-modal information fusion, characterized by, The system for implementing the method of any one of claims 1-7 comprises: A classification module: used for feature extraction, fusion, context disease representation, further encoding of disease state information, combination of disease state and disease name related information, and generation of rich disease embedding on the processed multi-modal data respectively; A generation module: taking the generated rich disease embedding as initial input, adopting a Transformer decoder with multi-head self-attention and position encoding, in the generation process, predicting the probability of the next word according to the previous word and the input embedding in the rich disease embedding, through the fusion of the word embedding and the disease embedding of the last step, gradually generating the text content of the medical report through multi-layer calculation and screening; An explanation module: used for checking and fine-tuning the text generated by the generation module, comparing and analyzing the text generated by the generation module with the disease-related theme generated by the classification module, checking whether the text content is consistent with the theme; if it is found that the generated text does not match the disease-related theme, the text is fine-tuned.

9. An electronic device, comprising: comprising: one or more processors; memory to store one or more programs; when the one or more programs are executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed, implement the method according to any one of claims 1-7.

Images