SOAP formatted medical record automatic generation method based on multi-modal dialogue

Abstract

The invention discloses an automatic SOAP formatted medical record generation method based on multi-modal dialogues, and relates to the technical field of artificial intelligence and medical information processing. The method aims at solving the problems that in the prior art, only text information is relied on, so that understanding is one-sided, and factual errors exist in large language model generation content. The method comprises the following steps: acquiring voice data of a doctor-patient dialogue and original text data converted from the voice data; respectively extracting acoustic rhythm features in the speech and semantic features in the text, and fusing the acoustic rhythm features and the semantic features into a unified multi-modal context representation; inputting the multi-modal context representation into a medical large language model, and dynamically weighting the generation probability of the large language model and the entity correlation score of the knowledge graph through a dynamic knowledge fusion gating mechanism by innovatively combining with the medical knowledge graph to obtain a multi-modal context representation; therefore, a formatted medical record text following an SOAP (subjective information S, objective information O, evaluation analysis A and a diagnosis and treatment plan P) structure is generated lexical element by lexical element. According to the method, the comprehensiveness of dialogue understanding is improved by fusing the multi-modal information, the professionality and the fact accuracy of the generated content are enhanced by utilizing the knowledge graph, and the automatic generation of the high-quality medical record is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to the interdisciplinary application of natural language processing and medical informatics.

[0002] More specifically, the present invention relates to a method for automatically generating SOAP-formatted medical records based on multimodal dialogue. Background Technology

[0003] In modern clinical practice, electronic medical records (EMRs) are the core of medical information management, while medical record writing is a time-consuming and arduous task in doctors' daily work. To improve clinical efficiency, voice input solutions based on automatic speech recognition (ASR) technology have emerged, which can transcribe doctor-patient conversations into text in real time, initially replacing traditional manual keyboard input. However, most of these solutions remain at the level of "speech-to-text," and the generated unstructured text still requires doctors to invest a lot of effort in manually sorting, summarizing, and filling it into standardized medical record templates (such as SOAP format), failing to fundamentally solve the burden of paperwork.

[0004] To achieve automated structure extraction, some methods attempt to utilize natural language processing techniques, such as keyword matching or rule templates, to extract fields like chief complaint and present medical history from transcribed text. However, these methods struggle to meet clinical requirements in terms of accuracy and generalization ability when faced with complex spoken expressions, contextual dependencies in multi-turn dialogues, and semantic ambiguity.

[0005] For example, Chinese patent application CN202511327349.0 discloses a method for generating structured electronic medical records based on speech recognition and a large language model. This method inputs text transcribed from speech into a large language model and guides the model to generate structured medical record fields through prompts. While this approach utilizes the powerful text understanding and generation capabilities of the large language model, it still has significant technical shortcomings. First, this method completely discards the original speech signal after speech recognition, using only plain text as the sole input for subsequent processing. This leads to the loss of crucial paralinguistic information, such as the intensity of the patient's tone when describing pain, the speed of speech, and pauses when hesitant. These acoustic prosodic features are crucial for accurately judging the patient's subjective feelings (S) and the severity of their condition, and cannot be fully captured by text content alone. Second, this method relies entirely on the parameterized knowledge within the large language model for content generation, lacking external, authoritative knowledge sources for fact-checking and guidance. This makes the model prone to "illusion" when generating professional content such as assessment analysis (A) and treatment plan (P), that is, fabricating inaccurate medical terminology, diagnosis or treatment plan, which poses potential risks to clinical application.

[0006] For example, Chinese patent application CN202210260578.5 discloses a method for constructing a natural language processing model based on big data. This method classifies text by constructing word vectors, calculating probability distributions, and using dendrograms for classification. This technical solution focuses on general text classification tasks, and its technical path is based on semantic analysis using statistical features and pre-defined structures, rather than being designed for generative tasks. This solution is not only completely limited to unimodal text processing and cannot be applied to multimodal scenarios such as doctor-patient dialogues, but its semantic analysis methods based on dendrograms and clustering are also limited in their ability to handle long-range dependencies, contextual understanding, and complex logical reasoning compared to modern large language models. Therefore, it is unsuitable for tasks such as generating logically rigorous and professionally content-driven SOAP format medical records.

[0007] In summary, existing technologies for automatically generating medical records from doctor-patient dialogues generally suffer from several technical problems: insufficient utilization of multimodal information within the dialogue, leading to incomplete understanding; and over-reliance on internal model knowledge, resulting in a failure to guarantee the medical professionalism and factual accuracy of the generated content. Therefore, there is an urgent need to propose a new technical solution that can deeply integrate the textual and acoustic information of the dialogue and introduce external medical knowledge to effectively constrain the generation process, thereby achieving end-to-end, high-fidelity, and highly reliable automatic generation of formatted medical records. Summary of the Invention

[0008] The purpose of this invention is to overcome the above-mentioned defects in the prior art and provide a method for automatically generating SOAP formatted medical records based on multimodal dialogue. This method aims to solve the technical problems of incomplete dialogue understanding due to reliance on text information alone and insufficient factuality of content generated by large language models due to a lack of external knowledge constraints.

[0009] To achieve the above objectives, this invention provides a method for automatically generating SOAP-formatted medical records based on multimodal dialogue, comprising: acquiring voice data of doctor-patient dialogue and raw text data converted from the voice data using a speech recognition model; extracting acoustic prosodic features and text semantic features from the voice data and the raw text data respectively to obtain multimodal features; fusing the acoustic prosodic features and the text semantic features to generate a unified multimodal context representation; inputting the multimodal context representation into a pre-trained medical large language model and combining it with a medical knowledge graph, and calculating the multimodal context representation at each generation time step using the following formula. candidate word groups final generation probability This allows for the generation of formatted medical record text following SOAP, namely, subjective information (S), objective information (O), assessment and analysis (A), and treatment plan (P) structure. , in, For the medical large language model at time step The context state, which is initialized by the multimodal context representation; Candidate nominatives generated by the medical big language model based on its internal parameters The base probability; In time step The set of medical entities retrieved from the medical knowledge graph that are relevant to the current context; Candidate word elements Relative to the set of medical entities Knowledge relevance score; For one in The dynamic knowledge fusion gating coefficient within the interval is used to dynamically weight the base probability and the knowledge relevance score.

[0010] Further, the step of fusing the acoustic prosodic features and the text semantic features specifically includes: using the text semantic features as a query vector and the acoustic prosodic features as a key vector and a value vector; calculating the attention weights of the text semantic features on the acoustic prosodic features through a cross-modal attention mechanism, and performing a weighted summation of the acoustic prosodic features to obtain a context-enhanced text representation that incorporates prosodic information; concatenating or adding the context-enhanced text representation with the original text semantic features to generate the unified multimodal context representation. The calculation process is shown in the following formula: , in, This is a matrix representation of the semantic features of the text. This is a matrix representation of the acoustic prosodic features; These are the learnable projection matrices for the query, key, and value, respectively; This represents the standard scaled dot product attention function; Presentation layer normalization operation.

[0011] Furthermore, the acoustic prosodic features include at least one extracted from the speech data: fundamental frequency profile, energy intensity, speech rate, or pause duration; the text semantic features are word vectors or sentence vector sequences obtained by inputting the raw text data into a pre-trained BERT model or a variant thereof.

[0012] Furthermore, the knowledge relevance score The calculation method is as follows: candidate word elements and the set of medical entities Each entity in All are mapped to the same high-dimensional vector space to obtain word vectors. and entity vector set ,in For entity collection The size; calculate the word vector With each entity vector The similarity between them is used; all similarity scores are normalized using the Softmax function to obtain the knowledge relevance score, the calculation process of which is shown in the following formula: , in, This is a function for calculating the cosine similarity of vectors. For the entity set Zhongyu Index of the most similar entity.

[0013] Furthermore, the dynamic knowledge fusion gating coefficient By a gating network based on the current context state The calculation process is shown in the following formula: , in, For the medical large language model at time step The hidden layer state vector; and These are the weight matrix and bias vector of the fully connected layer of the gated network, respectively. For the Sigmoid activation function, ensure The output value is between 0 and 1.

[0014] Furthermore, the set of medical entities retrieved from the medical knowledge graph The steps include: identifying existing medical concepts as seed entities from the currently generated medical record text and dialogue history; and in the medical knowledge graph, starting from the seed entities, using a breadth-first search or weighted random walk algorithm to retrieve neighboring entities within a predetermined number of hops to form the medical entity set. .

[0015] Furthermore, the step of generating formatted medical record text following the SOAP structure further includes: before the generation process begins, setting a structured control sequence, which sequentially contains four special control characters [S], [O], [A], and [P]; when the model generates a special control character, the text content generated subsequently is constrained within the medical record section represented by that control character, until the next special control character or end character is generated.

[0016] Furthermore, the medical big language model is obtained by enhancing domain knowledge and fine-tuning instructions based on a general big language model using a large amount of medical literature, electronic medical records and medical textbooks.

[0017] Furthermore, the word vector and entity vector It is obtained through a jointly trained embedding model that aims to minimize the distance between co-occurring lexical units and entities in the vector space in the medical corpus.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. Improved comprehensiveness and accuracy of information extraction: By integrating acoustic prosodic features and text semantic features in speech data, this invention can capture paralinguistic information such as tone, emotion, and stress that cannot be reflected in pure text, thereby gaining a more comprehensive understanding of the deeper meaning of doctor-patient dialogue and improving the accuracy of extracting subjective feelings (S) and objective signs (O).

[0019] 2. Enhanced medical professionalism and factual consistency in generated medical records: This invention innovatively introduces a medical knowledge graph to dynamically guide and constrain the generation process of the medical large language model. Through a dynamic knowledge fusion gating mechanism, authoritative entities in the knowledge graph are prioritized when generating key medical terms, diagnoses, and medication recommendations, effectively suppressing the "illusion" phenomenon of the model and ensuring the professionalism and factual accuracy of the assessment analysis (A) and treatment plan (P) sections.

[0020] 3. Achieves end-to-end automation and highly structured output: Through structured control sequences, this invention can automatically generate medical record texts that strictly adhere to the SOAP format, directly transforming unstructured doctor-patient dialogues into structured documents that are immediately usable in clinical workflows, greatly reducing doctors' paperwork burden and improving clinical work efficiency. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an overall flowchart of a SOAP-formatted medical record automatic generation method based on multimodal dialogue provided in an embodiment of the present invention.

[0023] Figure 2 This is a detailed flowchart of the multimodal feature fusion and knowledge graph guided generation module in this embodiment of the invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0026] Reference Figure 1 The present invention provides a method for automatically generating SOAP formatted medical records based on multimodal dialogue. Its overall process mainly includes core stages such as data acquisition and preprocessing, multimodal feature extraction, multimodal feature fusion, and knowledge graph-guided formatted text generation.

[0027] Step S101: Acquire multimodal dialogue data. In typical outpatient or remote consultation scenarios, the system uses deployed microphone devices to collect the complete dialogue between doctors and patients in real time, obtaining raw digital voice data. Simultaneously, a high-precision automatic speech recognition (ASR) model, such as an end-to-end model based on the Conformer or Transformer architecture, converts this voice data into timestamped raw text data, either in real time or offline.

[0028] Step S102: Extract multimodal features. This step extracts deep features in parallel from both speech and text modalities.

[0029] On the one hand, acoustic prosodic features are extracted from speech data. As described in claim 3, these features aim to quantify non-semantic information in speech. Specifically, open-source toolkits such as Praat or openSMILE can be used to perform frame-by-frame processing of the speech signal and extract a series of acoustic parameters frame by frame or syllable by syllable, including at least: fundamental frequency (F0) profile, reflecting changes in intonation; intensity, reflecting fluctuations in volume; speech rate, reflecting fluency of speech; and pause duration, reflecting hesitation or thinking in the dialogue. These features together constitute the acoustic prosodic feature sequence.

[0030] On the other hand, semantic features are extracted from the original text data. As described in claim 3, the text sequence transcribed from ASR is input into a language model pre-trained on a large medical corpus, such as PubMedBERT or BioClinicalBERT. This model is able to generate a high-dimensional vector representation containing rich contextual information for each token in the text; these vector sequences are the semantic features of the text.

[0031] Step S103: Perform multimodal feature fusion. To enable the model to comprehensively understand "what was said" (textual semantics) and "how it was said" (acoustic prosody), these two heterogeneous features must be effectively fused. (Refer to...) Figure 2 The fusion mechanism shown in this embodiment preferably employs a cross-modal attention mechanism, as defined in claim 2.

[0032] Specifically, the matrix representation of the text semantic feature sequence As a query vector, it represents the matrix of the acoustic prosodic feature sequence. As a key and value vector, by calculating attention weights, the model can adaptively learn the intrinsic association between specific words in the text (such as "pain," "difficulty breathing") and specific prosodic features in the speech (such as sudden pitch rises and slower speech rates). This fusion process can be represented by the following formula: , in, For text feature matrix, For acoustic feature matrix, and These represent sequence length and feature dimension, respectively. It is a learnable projection matrix used to map features from different modalities to a unified attention computation space. The function employs a standard scaled dot product attention mechanism. Finally, a unified multimodal context representation is generated through a residual connection and a layer normalization (LayerNorm) operation. This representation dynamically incorporates the most informative prosodic cues while preserving the original text's semantics.

[0033] Step S104: Generate SOAP-formatted medical records based on a medical big language model and knowledge graph. This step is the core innovation of this invention, as it integrates the multimodal context representation. As the initial context, it is input into a pre-trained medical large language model, which generates the target medical record text word by word in an autoregressive manner.

[0034] As described in claim 8, the medical large language model is preferably obtained by continuously training and fine-tuning instructions based on a general pre-trained large language model (such as the GPT series or LLaMA series models) using a large amount of professional language data, including medical literature, anonymized electronic medical records, medical textbooks, and clinical practice guidelines. This enables the model to not only master rich medical knowledge but also to understand and execute complex instructions (such as "generate SOAP format medical records").

[0035] To fundamentally address the inherent factual inconsistency (i.e., "illusion") problem in large language models and ensure the medical professionalism of the generated content, this invention introduces a dynamically guided generation mechanism driven by a medical knowledge graph. For example... Figure 2 As shown, at each generation time step The model does not directly use its internally calculated probability distribution, but rather determines the final lexical selection through a dynamic fusion process. Candidate lexical final generation probability Calculated using the following formula: , In this formula, It is a medical large language model at time step The internal context state (e.g., the hidden state vector at the top level of the Transformer decoder) carries all the generated information up to the current time step and the information generated by... Initialize multimodal input information. It is the basic probability distribution about the next lexical unit output by the large language model through its own softmax layer.

[0036] It is an external knowledge correction term, representing candidate lexical units. The system assesses the degree of relevance of medical knowledge to the current context. Its calculation first requires retrieving relevant entities from a medical knowledge graph. As described in claim 6, the system first identifies mentioned medical concepts (such as the disease "hypertension" and the drug "aspirin") as seed entities from the currently generated medical record text and the complete dialogue history using entity linking technology. Subsequently, in a large-scale medical knowledge graph (e.g., a graph integrating UMLS, SNOMED CT, drug knowledge bases, etc.), starting from these seed entities, a breadth-first search (BFS) or weighted random walk algorithm is used to retrieve neighboring entities within a predetermined number of hops (e.g., 1 or 2 hops). These entities collectively constitute the set of medical entities most relevant to the currently generated context. .

[0037] Next, as described in claim 4, a knowledge relevance score is calculated. First, each candidate lexicon in the vocabulary is... and entity set Each entity in The embedding model is mapped to the same high-dimensional vector space through a jointly trained embedding model. As described in claim 9, this embedding model, trained on a large medical corpus, ensures semantic consistency by bringing frequently co-occurring lexical units and entities closer together in the vector space. Then, candidate lexical unit vectors are calculated. With each entity vector The cosine similarity between the scores is calculated, and the Softmax function is used to normalize all similarity scores to obtain the final knowledge relevance score. The calculation process is as follows: , It is a dynamic knowledge fusion gating coefficient whose function is to intelligently determine, based on the current generation context, the extent to which to rely on the generative capabilities of the large language model and the extent to which to accept the precise facts of the knowledge graph. For example, when generating fluent narrative sentences or conjunctions, the model should rely more on its own language model. The value should approach 0; however, when generating a key diagnostic name, drug dosage, or anatomical term, it should rely more heavily on the knowledge graph. It should approach 1. As described in claim 5, this coefficient is determined by a lightweight gating network based on the current context state. Dynamic calculation yields: , in, and These are the trainable weight matrices and bias vectors of the fully connected layers in this gated network. It is the Sigmoid activation function, whose output value is constrained to... Within the range.

[0038] Finally, to ensure that the output text strictly adheres to the SOAP structure, as described in claim 7, this invention employs structured control characters for guidance. At the start of the generation task, an instruction prefix containing a special control character, such as "[S]", is input to the model. After the model generates this control character, all subsequent generated content is automatically marked as belonging to the "Subjective" section until the model generates the next control character (such as "[O]") or a text end marker. This mechanism imposes a rigid structural framework on the free text generation process, guaranteeing the format compliance of the final output.

[0039] For example, in a doctor-patient dialogue, the patient says, "I've had a headache lately, it's a throbbing pain." Voice analysis shows that when saying "throbbing pain," the patient's pitch and energy levels significantly increase. Traditional methods can only capture textual information, while the method of this invention, through multimodal fusion, can capture this emphasis on the nature of the pain. Therefore, when generating the subjective information (S) portion, it may generate "The patient complains of a headache, described as throbbing pain, with increased emphasis in the description," thus providing richer clinical clues.

[0040] When generating the assessment analysis (A) section, assuming the text has already mentioned "blood pressure 150 / 95 mmHg," the large language model itself might generate "consider hypertension." At this point, the knowledge graph retrieval module retrieves relevant entities based on "hypertension," such as "primary hypertension," "secondary hypertension," and "white coat hypertension." The knowledge fusion mechanism increases the probability of generating these specialized differential diagnostic terms. and through gating Integrating these findings into the final output makes the generated assessment analysis more rigorous and comprehensive. For example, "A: 1. High probability of primary hypertension; 2. White coat hypertension and secondary hypertension need to be ruled out." This effectively avoids diagnostic omissions caused by the model's knowledge limitations.

[0041] In summary, this invention achieves a deeper understanding of doctor-patient dialogue beyond the text level through deep fusion of multimodal features; significantly improves the professionalism and factual accuracy of generated medical record content through a dynamic collaborative generation mechanism of medical big language model and medical knowledge graph; and ensures that the output strictly conforms to SOAP format through structured control, ultimately providing an efficient, accurate, and automated medical record generation solution.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically generating SOAP-formatted medical records based on multimodal dialogue, characterized in that, include: Acquire voice data of doctor-patient dialogue, and raw text data converted from the voice data through a speech recognition model; Acoustic prosodic features and textual semantic features are extracted from the speech data and the original text data, respectively, to obtain multimodal features; The acoustic prosodic features and the text semantic features are fused to generate a unified multimodal context representation. This multimodal context representation is then input into a pre-trained medical large-scale language model and combined with a medical knowledge graph. The following formula is used to calculate the context representation at each generation time step. candidate word groups final generation probability This allows for the generation of formatted medical record text following SOAP, namely, subjective information (S), objective information (O), assessment and analysis (A), and treatment plan (P) structure. ， in, For the medical large language model at time step The context state, which is initialized by the multimodal context representation; Candidate nominatives generated by the medical big language model based on its internal parameters The base probability; In time step The set of medical entities retrieved from the medical knowledge graph that are relevant to the current context; Candidate word elements Relative to the set of medical entities Knowledge relevance score; For one in The dynamic knowledge fusion gating coefficient within the interval is used to dynamically weight the base probability and the knowledge relevance score.

2. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The step of fusing the acoustic prosodic features and the text semantic features specifically includes: using the text semantic features as a query vector and the acoustic prosodic features as a key vector and a value vector; calculating the attention weights of the text semantic features on the acoustic prosodic features through a cross-modal attention mechanism, and performing a weighted summation of the acoustic prosodic features to obtain a context-enhanced text representation that incorporates prosodic information; concatenating or adding the context-enhanced text representation with the original text semantic features to generate the unified multimodal context representation. The calculation process is shown in the following formula: ， in, This is a matrix representation of the semantic features of the text. This is a matrix representation of the acoustic prosodic features; These are the learnable projection matrices for the query, key, and value, respectively; This represents the standard scaled dot product attention function; Presentation layer normalization operation.

3. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The acoustic prosodic features include at least one of the following extracted from the speech data: fundamental frequency profile, energy intensity, speech rate, or pause duration; the text semantic features are sequences of word vectors or sentence vectors obtained by inputting the raw text data into a pre-trained BERT model or a variant thereof.

4. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The knowledge relevance score The calculation method is as follows: candidate word elements and the set of medical entities Each entity in All are mapped to the same high-dimensional vector space to obtain word vectors. and entity vector set ,in For entity collection The size; calculate the word vector With each entity vector The similarity between them is used; all similarity scores are normalized using the Softmax function to obtain the knowledge relevance score, the calculation process of which is shown in the following formula: ， in, This is a function for calculating the cosine similarity of vectors. For the entity set Zhongyu Index of the most similar entity.

5. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The dynamic knowledge fusion gating coefficient By a gating network based on the current context state The calculation process is shown in the following formula: ， in, For the medical large language model at time step The hidden layer state vector; and These are the weight matrix and bias vector of the fully connected layer of the gated network, respectively. For the Sigmoid activation function, ensure The output value is between 0 and 1.

6. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The set of medical entities retrieved from the medical knowledge graph The steps include: identifying existing medical concepts as seed entities from the currently generated medical record text and dialogue history; and in the medical knowledge graph, starting from the seed entities, using a breadth-first search or weighted random walk algorithm to retrieve neighboring entities within a predetermined number of hops to form the medical entity set. .

7. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The step of generating formatted medical record text following the SOAP structure further includes: before the generation process begins, setting a structured control sequence, which sequentially contains four special control characters [S], [O], [A], and [P]; when the model generates a special control character, the text content generated subsequently is constrained within the medical record section represented by that control character until the next special control character or end character is generated.

8. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 1, characterized in that, The medical big language model is obtained by enhancing domain knowledge and fine-tuning instructions based on a general big language model using a large amount of medical literature, electronic medical records and medical textbooks.

9. The method for automatically generating SOAP-formatted medical records based on multimodal dialogue according to claim 4, characterized in that, The lexical vector and entity vector It is obtained through a jointly trained embedding model that aims to minimize the distance between co-occurring lexical units and entities in the vector space in the medical corpus.

Images