Doctor-patient dialogue real-time structured case generation system and method based on end-to-end semantic understanding

Abstract

The invention specifically relates to a doctor-patient dialogue real-time structured case generation system and method based on end-to-end semantic understanding, and relates to the technical field of medical artificial intelligence and deep learning, and the method comprises the steps: extracting a high-dimensional acoustic feature sequence; inputting the high-dimensional acoustic feature sequence into a preset decoder model, and performing autoregression decoding on the acoustic feature sequence; and generating a text sequence which is in a predefined format and contains the structured medical record content. In the invention, the end-to-end model constructs direct mapping between the acoustic mode and the medical semantic unit, and audio does not need to be transwritten into characters, so that the problem of downward accumulation of ASR (Accelerated Solvent Response) transwriting errors in a traditional scheme is eradicated, the accuracy and reliability of medical record contents are remarkably improved, surface pronunciation differences can be spanned, and the accuracy and the reliability of the medical record contents are improved. Common acoustic features of different dialects in context and dialogue logic are captured, robustness to complex acoustic environments such as consulting room noise and dialects is greatly enhanced, and diversified diagnosis and treatment scene requirements are met.

Description

Technical Field

[0001] This invention relates to the fields of medical artificial intelligence and deep learning technology, and in particular to a system and method for generating real-time structured medical records based on end-to-end semantic understanding of doctor-patient dialogue. Background Technology

[0002] In modern healthcare systems, outpatient doctors need to quickly obtain subjective information about a patient's condition through dialogue, combine this with objective examinations to make diagnostic inferences, and ultimately form treatment decisions. At the same time, laws, regulations, and medical standards require doctors to record this entire diagnostic and treatment process accurately and in standardized language as a legally binding electronic medical record (EMR).

[0003] However, there is an irreconcilable conflict of time and energy between doctors and the two core tasks of communicating with patients and inputting data into machines, including: A significant amount of clinical time is consumed in writing and entering medical records. Doctors need to listen to patients' spoken, fragmented, and even dialect-influenced statements while simultaneously filtering, reorganizing, and translating the information into written language that conforms to medical standards in their minds, and then inputting it word by word into the EMR system via the keyboard. During the diagnosis and treatment process, doctors frequently look at the computer screen instead of the patient; patients do not feel the doctor's attention, making it difficult to build trust, and their medical experience and satisfaction are greatly reduced; while doctors may miss some non-verbal diagnostic clues because they cannot make sufficient eye contact and observe body language.

[0004] To address the aforementioned issues, the industry has long explored solutions for voice-based medical record entry, including: Automatic speech recognition (ASR) is the process of transcribing audio of doctor-patient conversations into plain text.

[0005] Natural Language Understanding (NLU) is a process that uses a separate model to extract information from transcribed text and populate it into a medical record template.

[0006] However, traditional solutions have fundamental flaws that are difficult to overcome: ASR (Automatic Speech Recognition) is the first step in the entire process, and its accuracy is crucial. However, in a real clinic environment, due to background noise, variations in speech rate, and especially the presence of complex dialects and accents, transcription errors in ASR are inevitable. In addition to textual content, speech signals also contain rich paralinguistic information such as emotion, stress, speech rate, and hesitation, which are lost when speech is converted into text.

[0007] The success or failure of the entire system hinges on the extremely difficult task of achieving perfect text transcription, especially in the multi-dialect-speaking Chinese medical environment.

[0008] The emergence of end-to-end deep learning models offers a revolutionary new approach to completely overturn the aforementioned dilemmas. In other fields, such as machine translation, end-to-end models have proven that by constructing a unified model that directly maps source language sentences to target language sentences, better results can be achieved than the traditional method of first analyzing the syntax tree and then generating the syntax tree.

[0009] Therefore, a real-time structured case generation system and method for doctor-patient dialogue based on end-to-end semantic understanding is proposed to address the aforementioned problems. Summary of the Invention

[0010] The purpose of this invention is to address the aforementioned problems by proposing a real-time structured case generation system and method for doctor-patient dialogue based on end-to-end semantic understanding.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: A real-time structured medical record generation system for doctor-patient dialogue based on end-to-end semantic understanding includes: Acquire the audio stream containing the dialogue between the doctor and the patient, identify the person corresponding to each speech segment in the audio stream, and input the identification information as an additional feature into the end-to-end model; The construction of an end-to-end speech and language understanding model includes: The dialogue audio stream is input into a pre-trained acoustic encoder model to extract a high-dimensional acoustic feature sequence. The high-dimensional acoustic feature sequence is input into the preset decoder model, and the acoustic feature sequence is subjected to autoregressive decoding. Generate a predefined formatted text sequence containing structured medical record content, which includes at least the chief complaint, present illness, and past medical history.

[0012] Preferably, after acquiring the audio stream containing the doctor and patient's dialogue, the process further includes noise reduction and speech enhancement preprocessing of the audio stream.

[0013] Preferably, the pre-trained acoustic encoder model is a Wav2Vec2.0 model or a HuBERT model, which has been pre-trained using self-supervised learning on a large-scale unlabeled speech dataset.

[0014] Preferably, the text sequence generated by the decoder follows a preset markup language format, which includes special structural markers for defining the medical record structure and natural language vocabulary for representing the content.

[0015] Preferably, the method further includes: Construct a parallel corpus containing a large amount of doctor-patient dialogue audio and its corresponding manually annotated, structured target medical record data; By using parallel corpora, the end-to-end model consisting of an acoustic encoder and decoder is fine-tuned through supervised learning to build a direct mapping relationship between the model learning the original dialogue audio and the target structured medical record content.

[0016] Preferably, the parallel corpus contains a preset number of regional dialect audio data to enhance the robustness of the end-to-end model to different dialects.

[0017] Preferably, the decoder model generates at least one confidence score while generating the text sequence, which represents the degree of certainty of the model regarding each part of the structured medical record content it generates.

[0018] Preferably, the method further includes: The text sequence generated by the decoder is parsed to extract structured medical record data, and this data is automatically populated into the corresponding medical record chapter field in the electronic medical record system. The populated content is presented to the doctor user in a visual format on the user interface.

[0019] A method for generating real-time structured medical records based on end-to-end semantic understanding of doctor-patient dialogues includes the following components: Information Acquisition and Preprocessing: Acquire the audio stream of doctor-patient dialogue, perform noise reduction and speech enhancement preprocessing on it, and send the processed dialogue audio stream to the end-to-end speech and language understanding model; End-to-end speech and language understanding model construction: Receive the processed dialogue audio stream, extract the high-dimensional acoustic feature sequence, and generate a predefined text sequence containing structured medical record content from the high-dimensional acoustic feature sequence; Data parsing and electronic medical record integration: Parse the text sequence generated by the decoder, extract the structured medical record data, and automatically populate it into the corresponding fields of the electronic medical record system; User interaction and review: The filled content is presented to the doctor user in a visual state pending confirmation on the user interface, so that the doctor can make final review, modification and confirmation.

[0020] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention constructs a direct mapping between acoustic patterns and medical semantic units through an end-to-end model, eliminating the need to first transcribe audio into text. This not only eradicates the problem of ASR transcription errors accumulating downwards in traditional solutions, significantly improving the accuracy and reliability of medical record content, but also transcends surface pronunciation differences, capturing the common acoustic features of different dialects in context and dialogue logic. This greatly enhances robustness to complex acoustic environments such as clinic noise and dialects, adapting to diverse diagnostic and treatment scenarios.

[0021] 2. This invention uses an end-to-end model to directly map acoustic patterns to medical semantic units, eliminating the need to first transcribe audio into text. Instead, it directly identifies medical concepts such as headaches and pain as colic. This not only eliminates the problem of ASR transcription errors accumulating downwards in traditional solutions, significantly improving the accuracy and reliability of medical record content, but also transcends surface pronunciation differences, capturing common acoustic features of different dialects in context and dialogue logic. This greatly enhances robustness to complex acoustic environments such as clinic noise and dialects, adapting to diverse diagnostic and treatment scenarios. Attached Figure Description

[0022] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a system structure diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0023] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0024] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0025] Example 1 Its specific implementation method is combined with the appendix Figure 1 and attached Figure 2 Please provide a detailed explanation.

[0026] Appendix Figure 1 The block diagram of the real-time structured medical record generation system based on end-to-end semantic understanding for doctor-patient dialogue provided in the embodiments of the present invention shows the connection relationship between acquiring the audio stream of the dialogue between the doctor and the patient and generating a text sequence containing structured medical record content in a predefined format.

[0027] Appendix Figure 2 The flowchart of the real-time structured case generation method for doctor-patient dialogue based on end-to-end semantic understanding provided in the embodiments of the present invention illustrates the complete steps from information acquisition and preprocessing to user interaction and review.

[0028] In this embodiment, it includes: Using at least one audio acquisition device, the audio stream containing the dialogue between the doctor and the patient is acquired in real time. The identities of the people corresponding to each speech segment in the audio stream are marked, and the identity marking information is input into the end-to-end model as an additional feature. After acquiring the audio stream containing the doctor's and patient's dialogue, the process also includes noise reduction and speech enhancement preprocessing of the audio stream, specifically: Beamforming and acoustic echo cancellation (AEC) techniques are employed to denoise and enhance the speech of the dialogue audio stream. Preprocessing the audio stream with denoising and speech enhancement effectively filters out background noise in the clinical environment (such as instrument sounds and background conversations), improves the clarity of the original audio, provides high-quality input for subsequent acoustic feature extraction by the model, and reduces noise interference with feature recognition.

[0029] Preprocessed audio can enhance weak speech signals (such as patients whispering their symptoms), ensuring that key medical information is not drowned out, especially in noisy outpatient settings, and ensuring that the model can fully capture the content of doctor-patient dialogue, laying the foundation for accurate medical record generation.

[0030] By identifying and labeling voice segments in the audio stream of doctor-patient dialogues and using them as feature inputs, the model can clearly distinguish between doctor's consultations and patient statements, helping it to accurately understand the dialogue logic and information attribution, avoid misjudging medical record content due to role confusion, and improve the accuracy of capturing key information in doctor-patient interactions.

[0031] The end-to-end model directly extracts features from the audio stream and generates structured medical records containing core content such as chief complaint and present medical history. It eliminates the multi-step conversion from traditional speech to text to text parsing, reduces the accumulation of intermediate errors, and achieves efficient mapping from dialogue to medical records, meeting the needs of real-time clinical recording.

[0032] The construction of the end-to-end speech and language understanding model (the end-to-end speech and language understanding model is obtained through supervised learning fine-tuning on a proprietary medical corpus containing a large number of data pairs (e.g., audio, structured medical records)) specifically includes: The dialogue audio stream is input into a pre-trained acoustic encoder model to extract a high-dimensional acoustic feature sequence that can represent the content of the dialogue audio; wherein, the pre-trained acoustic encoder model is a Wav2Vec2.0 model or a HuBERT model, which has been pre-trained in self-supervised learning on a large-scale unlabeled speech dataset. Acoustic encoders such as Wav2Vec2.0 or HuberT, which are pre-trained on large-scale unlabeled speech data, have learned deep feature patterns of general speech. They do not need to be trained from scratch and can be quickly adapted to medical scenarios, reducing model development costs and improving feature extraction efficiency.

[0033] These models can accurately capture details such as the pronunciation of professional terms and changes in tone in medical dialogues, generating high-dimensional acoustic features that are more in line with medical scenarios, providing a reliable foundation for decoders to generate accurate structured medical records.

[0034] A high-dimensional acoustic feature sequence is input into a pre-trained decoder model (based on a Transformer architecture), and the acoustic feature sequence is subjected to autoregressive decoding. The text sequence generated by the decoder follows a pre-defined markup language format, which includes special structural tags for defining the medical record structure (such as chapters and fields) and natural language vocabulary for representing the content. While generating the text sequence, the decoder model also generates at least one confidence score to indicate the degree of certainty of the model for each part of the structured medical record content it generates.

[0035] The confidence score generated by the decoder can quantify the certainty of each part of the medical record content, helping doctors quickly locate low-confidence content (such as vague symptom descriptions), focus on the key points of review, reduce repeated checks on high-certainty content, and improve review efficiency.

[0036] Confidence score provides an intuitive reference for the reliability of the system output. Doctors can judge the credibility of the content based on the score, reducing the risk of medical record errors caused by model misjudgment, and ensuring the accuracy of clinical records while improving the degree of automation.

[0037] The text generated by the decoder follows a format with special structural markers, which can clearly distinguish the chapters (such as chief complaint and present illness history) and field boundaries of the medical record, making the generated content conform to the medical document standard, facilitating subsequent automatic parsing and structured processing by the system, and reducing format confusion issues.

[0038] By combining natural language vocabulary, the system maintains semantic integrity while ensuring structural rigor. This satisfies the requirements of electronic medical record systems for structured data, facilitates doctors' intuitive understanding of the content, and balances machine processing efficiency with human readability.

[0039] Model fine-tuning: Construct a parallel corpus containing a large amount of doctor-patient dialogue audio and its corresponding manually annotated, structured target medical record data; By utilizing parallel corpora, the end-to-end model consisting of an acoustic encoder and decoder is fine-tuned through supervised learning to build a direct mapping relationship between the model learning the original dialogue audio and the target structured medical record content. The parallel corpora contain a preset number of regional dialect audio data to enhance the robustness of the end-to-end model to different dialects.

[0040] Constructing parallel corpora of doctor-patient dialogue audio and structured medical records provides targeted training data for model fine-tuning, enabling the model to learn the correspondence between voice content and medical record elements specific to medical scenarios, thereby improving the clinical adaptability of the generated content.

[0041] By fine-tuning and strengthening the end-to-end mapping between the acoustic encoder and decoder through supervised learning, the model can directly master the conversion logic from raw audio to target medical records, reduce errors caused by multi-module serialization, and make the generated structured medical records more in line with actual diagnosis and treatment needs.

[0042] The text sequence generated by the decoder is parsed to extract structured medical record data, and this data is automatically populated into the corresponding medical record chapter field in the electronic medical record (EMR) system. The filled content is presented to the doctor user in a visual state pending confirmation, so that the doctor can make final review, modification and confirmation.

[0043] Automatically parses text sequences and fills them into the corresponding fields of the electronic medical record system, saving doctors the work of manual input and format adjustment, greatly reducing the time spent writing medical records, allowing doctors to focus more on diagnosis and treatment, and improving the efficiency of outpatient work.

[0044] Visual presentation of the filled content makes it easy for doctors to view, modify, and confirm, ensuring that the medical record content is consistent with the actual diagnosis and treatment; at the same time, it achieves seamless integration with the electronic medical record system, promotes the standardized management and flow of diagnosis and treatment data, and optimizes the overall diagnosis and treatment process.

[0045] Example 2 Please see Figure 2 A method for generating real-time structured medical records for doctor-patient dialogues based on end-to-end semantic understanding includes the following components: Information Acquisition and Preprocessing: Acquire the audio stream of doctor-patient dialogue, perform noise reduction and speech enhancement preprocessing on it, and send the processed dialogue audio stream to the end-to-end speech and language understanding model; End-to-end speech and language understanding model construction: Receive the processed dialogue audio stream, extract the high-dimensional acoustic feature sequence, and directly autoregressively generate a predefined format text sequence containing structured medical record content from the high-dimensional acoustic feature sequence; Data parsing and electronic medical record integration: Parse the text sequence generated by the decoder, extract the structured medical record data, and automatically populate it into the corresponding fields of the electronic medical record system; User interaction and review: The filled content is presented to the doctor user in a visual state pending confirmation on the user interface, so that the doctor can make final review, modification and confirmation.

[0046] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0047] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0048] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0049] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0050] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0051] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0052] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0053] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0054] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0055] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A doctor-patient conversation real-time structured case generation system based on end-to-end semantic understanding, characterized in that, The method comprises the following steps: Obtain the dialogue audio stream containing the doctor and the patient, and mark the identities of the persons in each speech segment in the audio stream, and input the identity marking information as additional features into the end-to-end model; The end-to-end speech language understanding model is constructed, specifically comprising: Input the dialogue audio stream into a pre-trained acoustic encoder model to extract a high-dimensional acoustic feature sequence; Input the high-dimensional acoustic feature sequence into a preset decoder model and perform autoregressive decoding on the acoustic feature sequence; Generate a text sequence in a predefined format containing structured medical record content, which at least includes chief complaint, history of present illness and past history. 2.The doctor-patient conversation real-time structured case generation system based on end-to-end semantic understanding according to claim 1, wherein, After obtaining the dialogue audio stream containing the doctor and the patient, the dialogue audio stream is also preprocessed by noise reduction and speech enhancement. 3.The doctor-patient conversation real-time structured case generation system based on end-to-end semantic understanding according to claim 1, wherein, The pre-trained acoustic encoder model is a Wav2Vec2.0 model or a HuBERT model, which has been pre-trained through self-supervised learning on a large-scale unlabeled speech data set. 4.The end-to-end semantic understanding based doctor-patient conversation real-time structured case generation system according to claim 1, wherein, The text sequence generated by the decoder follows a preset markup language format, which contains special structure markers for defining medical record structure and natural language vocabulary for representing content. 5.The end-to-end semantic understanding based doctor-patient conversation real-time structured case generation system according to claim 1, wherein, Further comprising: Construct a parallel corpus containing a large amount of doctor-patient dialogue audio and its corresponding artificially annotated structured target medical record data; Use the parallel corpus to fine-tune the end-to-end model composed of the acoustic encoder and the decoder through supervised learning, so as to construct the model to learn the direct mapping relationship from the original dialogue audio to the target structured medical record content. 6.The end-to-end semantic understanding based doctor-patient conversation real-time structured case generation system according to claim 5, characterized in that, The parallel corpus contains a preset amount of regional dialect audio data to enhance the robustness of the end-to-end model to different dialects. 7.The end-to-end semantic understanding based doctor-patient conversation real-time structured case generation system according to claim 1, wherein, The decoder model generates at least one confidence score while generating the text sequence, which represents the certainty degree of the model for each part of the structured medical record content it generates. 8.The end-to-end semantic understanding based doctor-patient conversation real-time structured case generation system according to claim 1, wherein, Further comprising: Parse the text sequence generated by the decoder to extract structured medical record data, and automatically fill the data into the corresponding medical record chapter field of the electronic medical record system; Present the filled content to the doctor user in a visual state on the user interface.

9. The method according to any one of claims 1-8, wherein the system is characterized in that, The method comprises the following parts: Information acquisition and preprocessing: obtain the doctor-patient dialogue audio stream, and perform noise reduction and speech enhancement preprocessing on it, and send the processed dialogue audio stream to the end-to-end speech language understanding model; End-to-end speech language understanding model construction: receive the processed dialogue audio stream, extract a high-dimensional acoustic feature sequence, and generate a text sequence in a predefined format containing structured medical record content from the high-dimensional acoustic feature sequence; Data parsing and electronic medical record integration: parse the text sequence generated by the decoder to extract structured medical record data, and automatically fill it into the corresponding field of the electronic medical record system; User interaction and review: present the filled content to the doctor user in a visual state to be confirmed on the user interface for final review, modification and confirmation.

Images