Multi-mode speech recognition and structured electronic medical record generation system based on environmental perception

Abstract

According to the multi-mode speech recognition and structured electronic medical record generation system based on environmental perception, the dialogue content between a doctor and a physical examination person and the oral instruction / result of the doctor are accurately captured in a non-inductive mode and are intelligently filled into the predefined structured electronic medical record template of the corresponding department, so that the manual input time of the doctor is shortened, and the efficiency is improved. The system liberates both hands and eyes, focuses on inquiry and examination, improves the physical examination quality, and improves the satisfaction degree of physical examinees.

Description

Technical Field

[0001] This invention relates to the field of hospital information technology, and in particular to an environment-aware multimodal speech recognition and structured electronic medical record generation system and its application method, electronic equipment and computer-readable storage medium. Background Technology

[0002] During consultations and examinations, doctors at health checkup centers need to communicate with patients, operate equipment, and simultaneously enter test results into a computer. This leads to: a. Inefficiency: Doctors' attention is scattered, data entry takes a long time, and the flow rate of physical examinations is affected; b. Prone to errors: Manual data entry is prone to typos, omissions, or incorrect selections; c. Poor experience: Doctors are busy with data entry, which affects the communication experience with examinees, and examinees' waiting time is also extended; d. Difficulty in standardization: Different doctors have different data entry habits, which affects data quality and subsequent analysis; 2. Existing speech recognition systems are mostly designed for general scenarios and have not been optimized for the professional terminology of physical examination departments (such as "refractive error" and "tonsil enlargement") or doctor-patient dialogue modes (such as "Do you have a family history?" and "Is your blood pressure high?"), resulting in low recognition accuracy.

[0003] Although existing health checkup centers have fixed departments, relatively standardized examination items, and mature data entry templates, they lack an intelligent means to automatically, accurately, and efficiently convert doctors' natural language communication and oral statements in the consultation room into structured electronic medical record data. Summary of the Invention

[0004] To address the technical problems existing in the prior art, the present invention provides the following technical solution, used for... On the one hand, a system for generating structured electronic medical records based on environment-aware multimodal speech recognition is provided, the system comprising: The environmental perception module is used to perceive and determine the examinee's ID and current department / clinic in real time; A multimodal speech acquisition and preprocessing module is used to acquire and preprocess speech within the clinic. The Natural Language Understanding and Structured Processing module is used to convert speech into structured information that matches the department's professional terminology; A structured electronic medical record template library is used to store predefined structured templates for each department; The structured template matching and autofill module is used to map and fill structured information into the corresponding structured template; The human-computer interaction and verification module is used by doctors to review and correct medical records; The data storage and interface module is used for storing medical records and connecting with the HIS (Hospital Information System).

[0005] Preferably, the environmental perception module includes: The clinic-department identification unit is used to match the clinic number through a preset "clinic-department" mapping database, or to double-confirm the department corresponding to the current clinic by combining the department code logged in by the doctor. The examinee location and identification unit is used to determine the examinee's current examination room through RFID / Bluetooth beacons or computer vision technology.

[0006] Preferably, the multimodal speech acquisition and preprocessing module includes: Microphone arrays are used to capture the voices of doctors and patients undergoing physical examinations and to distinguish the direction of the speaker. A multimodal speech recognition engine is used to include a high-performance automatic speech recognition (ASR) sub-engine (which converts speech streams to text in real time), a speaker separation and role recognition sub-engine (which distinguishes doctor / examinee speech based on voiceprint, location or preset role), a context-aware sub-engine (which optimizes the recognition of contextual terms by combining departmental environment, examination equipment or template fields) and / or an instruction recognition sub-engine (which recognizes doctor instructions).

[0007] Preferably, the natural language understanding and structuring module includes: The NLP model operation module is used to train and optimize NLP models based on the professional terminology, common chief complaints, examination items and result descriptions of various departments in the health checkup center. The information extraction submodule is used to call NLP models to identify key entities, semantic relationships, and distinguish between declarative and interrogative sentences. The intent recognition submodule is used to determine the doctor's speech intent; The dialect / accent adaptation submodule is used to adapt to the dialects of the age groups of people undergoing physical examinations.

[0008] Preferably, the structured template matching and auto-fill module includes: The template matching unit is used to automatically load the structured template of the corresponding department based on the department information provided by the environmental perception module. Field mapping unit, used to map structured information to corresponding fields in the template through exact matching, semantic matching or contextual association; The autofill cell is used to fill numerical values ​​into corresponding numerical fields, match standardized descriptions to options or fill them into description boxes, or fill free text into corresponding text boxes.

[0009] Preferably, the human-computer interaction and verification module includes: The real-time processing module is used to generate preliminary medical record drafts during or shortly after consultations. The voice / touch correction module allows doctors to modify draft medical records via voice commands or touch operations; The key confirmation point module is used to provide voice prompts or pop-up summaries for doctors to confirm when completing an examination item or ending an examination in the clinic. The HIS integration module is used to import structured data into the HIS system's medical record templates with one click or automatically.

[0010] Preferably, the medical record data storage is integrated with the HIS, including: The confirmed electronic medical records will be stored in the hospital system; It supports integration with systems such as HIS / LIS to enable data exchange.

[0011] On the other hand, an application method for an environment-aware multimodal speech recognition and structured electronic medical record generation system is provided, the method comprising: Real-time sensing and determination of the examinee's ID and current department / clinic; determining the patient's department based on the sensing information and calling the corresponding department's structured template; Collect and preprocess speech within the clinic; The speech is converted into structured information that matches the department's professional terminology and mapped and filled into the corresponding structured template, generating the corresponding medical record draft and sending it to the human-computer interaction and verification module; Doctors review and correct medical records through the human-computer interaction and verification module. Once the review is complete, the medical records are stored in the HIS.

[0012] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the application method of the environment-aware multimodal speech recognition and structured electronic medical record generation system described above is implemented.

[0013] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-described application method of the environment-aware multimodal speech recognition and structured electronic medical record generation system.

[0014] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a block diagram of a system for generating structured electronic medical records based on environmental perception multimodal speech recognition, provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the training process of an NLP model provided in an embodiment of the present invention; Figure 3 This is an application flowchart of an environment-aware multimodal speech recognition and structured electronic medical record generation system provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0019] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0020] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0021] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0022] This invention provides a system for generating structured electronic medical records based on environment-aware multimodal speech recognition, such as... Figure 1 The diagram shown is a block diagram of an environment-aware multimodal speech recognition and structured electronic medical record generation system. The system includes: The environmental perception module is used to perceive and determine the examinee's ID and current department / clinic in real time; A multimodal speech acquisition and preprocessing module is used to acquire and preprocess speech within the clinic. The Natural Language Understanding and Structured Processing module is used to convert speech into structured information that matches the department's professional terminology; A structured electronic medical record template library is used to store predefined structured templates for each department; The structured template matching and autofill module is used to map and fill structured information into the corresponding structured template; The human-computer interaction and verification module is used by doctors to review and correct medical records; The data storage and interface module is used for storing medical records and connecting with the HIS (Hospital Information System).

[0023] The working principle of each module will be described in detail below.

[0024] 1. Environmental Perception Module This module aims to determine the identity of the examinee and their department / clinic in real time. By automatically identifying the department corresponding to the clinic and locating and identifying the examinee, it achieves accurate perception of the examinee's location and identity in the medical examination environment.

[0025] (1) Automatically identify the department corresponding to the current consultation room A pre-defined "clinic-department" mapping database is provided. This database has a fixed configuration but supports background updates. The system will automatically match the corresponding department information in the database based on the clinic number. An alternative approach is to combine the department code that doctors log in to on the computer in the clinic. Since the department code is linked to the department, this double verification of department affiliation improves the accuracy of identification.

[0026] (2) Physical examination examinee location and identification unit Option A (RFID / Bluetooth Beacon): The examinee must wear a wristband / card containing an RFID tag or Bluetooth beacon. Readers / beacons are deployed at the entrance or inside the examination room. When an examinee enters a specific examination room, the reader / beacon can identify the tag or beacon they are carrying, thus confirming that the examinee has entered that examination room.

[0027] Option B (Computer Vision): Install cameras in the examination room (privacy issues need to be considered) or place sensors at the entrance to identify examinees entering. Combine this with facial recognition (with the examinee's authorization) or wristband identification to confirm the examinee's identity.

[0028] For example, in a certain health checkup center, each consultation room has a unique number, such as "General Practice Room 1" and "Ophthalmology Room 2." The system has a pre-set "consultation room-department" mapping database. When a doctor logs in on the computer in the consultation room, the department code they log in with is also recorded. When a person undergoing health checkup enters "Ophthalmology Room 2" wearing a wristband with an RFID tag, the RFID reader at the door identifies the tag information. At the same time, the system matches the consultation room number from the database to find the corresponding department, which is "Ophthalmology." This is then combined with the department code logged in by the doctor for double confirmation, ultimately confirming that the person undergoing health checkup is in the "Ophthalmology" consultation room.

[0029] This module enables real-time and accurate acquisition of the examinee's location and identity information, providing foundational data for subsequent modules such as voice acquisition and medical record generation, ensuring the efficiency and accuracy of the entire physical examination process.

[0030] 2. Multimodal speech acquisition and preprocessing module This module is mainly responsible for collecting the voices of doctors and patients in the clinic and preprocessing the collected voices, including noise reduction, sound source localization, speech recognition, speaker separation, context awareness and instruction recognition, so as to facilitate subsequent natural language understanding and structured processing.

[0031] Microphone arrays: Deploy high-sensitivity microphone arrays with noise reduction and sound source localization capabilities within the examination room, such as ceiling-mounted microphones. These microphones effectively capture the voices of doctors and patients, suppress ambient noise through noise reduction, and use sound source localization to distinguish the speaker's direction.

[0032] (1) Multimodal speech recognition engine: A high-performance automatic speech recognition engine is used to convert the collected speech stream into text in real time.

[0033] Speaker separation and role recognition engine: Based on information such as voiceprint, location or preset role, it identifies and distinguishes the voice of doctors and examinees.

[0034] Context-aware engine: Optimizes recognition performance by combining the current departmental environment, the currently operating examination equipment (if integrated with equipment interfaces), and pre-loaded template fields. For example, in an ophthalmology clinic, it prioritizes the recognition of terms such as "visual acuity," "intraocular pressure," and "fundus."

[0035] Command recognition engine: Recognizes specific commands issued by doctors, such as "Record: Blood pressure 120 / 80 mmHg" or "Chief complaint: Intermittent headache for one week".

[0036] In the ophthalmology clinic, a ceiling-embedded microphone array captures the voices of both the doctor and the patient. The core Automatic Speech Recognition (ASR) function converts the speech to text in real time, while speaker separation and role recognition distinguish the doctor's and patient's voices based on voiceprint and location information. Because it's an ophthalmology clinic, the context-aware function prioritizes recognizing ophthalmology-related terms; for example, if the doctor says "Check my eye pressure," the system can accurately recognize it. Simultaneously, when the doctor issues the instruction "Record: Eye pressure 15 mmHg," the instruction recognition function can identify the command.

[0037] This module can collect clear and accurate speech information and perform effective preprocessing, thereby improving the accuracy and efficiency of speech recognition and providing high-quality text data for subsequent natural language understanding and structured processing.

[0038] 3. Natural Language Understanding and Structured Processing Module The main task of this module is to accurately convert speech into text and adapt it to the professional terminology of various departments. By training a domain-specific NLP model, it performs information extraction and intent recognition on the text, transforming unstructured text into structured data for subsequent medical record generation and management.

[0039] Domain-specific NLP models: These models are trained and optimized based on the professional terminology, common complaints, examination items, and result descriptions of various departments within a health checkup center, enabling them to better understand and process textual information from each department. The training process for these NLP models is as follows (combined with...). Figure 2 (as shown) (1) Data collection Technical Terminology Collection: Conduct thorough research across all departments within the health checkup center to collect specialized medical terminology. For example, in the laboratory, collect terms like "eosinophils" and "lymphocyte percentage" from routine blood tests; in the radiology department, collect professional descriptions such as "linear opacities in the lungs" and "low-density lesions in the liver." These terms can be obtained by consulting departmental diagnostic manuals, medical records, and examination report templates.

[0040] Common chief complaints collection: Collect common chief complaints from the medical records of examinees. For example, common complaints in internal medicine include "dizziness and fatigue," while common complaints in surgery include "joint pain and limited mobility." All medical records from examinees can be collected over a period of time (e.g., one month), then organized and categorized.

[0041] Examination Item Collection: Compile a list of examination items for each department. The laboratory department offers blood biochemistry tests (including blood glucose, blood lipids, liver function, etc.) and routine urine tests; the radiology department offers X-ray examinations, CT scans, and ultrasound examinations. This information can be obtained from the health checkup center's billing system and examination appointment system.

[0042] Results Description Collection: This involves collecting descriptive texts of examination results from various departments. For example, electrocardiogram (ECG) results might be described as "sinus arrhythmia" or "ST-T segment changes"; ophthalmological examination results might be described as "increased myopia" or "retinal lesions." These results descriptions are extracted from a large number of historical examination reports.

[0043] (2) Data preprocessing Data cleaning: Remove noise from the collected data, such as duplicate records, incorrect punctuation, and non-standard characters. For example, remove extra parentheses and spaces from "Blood pressure: 120 / 80mmHg (normal)" to get "Blood pressure: 120 / 80mmHg".

[0044] Data annotation: Annotate the data to facilitate model learning. For technical terms, you can annotate the department and category to which they belong; for common complaints, you can annotate the possible corresponding diseases or health problems; for examination items, you can annotate the examination site and purpose; for result descriptions, you can annotate whether they are normal or abnormal and the possible diagnostic directions. For example, "dizziness and fatigue" can be annotated as "internal medicine, possibly related to anemia and low blood pressure".

[0045] Word segmentation: Use appropriate word segmentation tools to segment the text. For Chinese text, tools such as Jieba can be used. For example, "lung cord-like shadow" can be segmented into "lung", "cord-like", and "shadow".

[0046] (3) Model selection and training Choosing a suitable model architecture: You can choose a pre-trained NLP model, such as BERT or RoBERTa, and fine-tune it. These models have powerful language understanding capabilities and can learn the semantic information of text.

[0047] Dataset partitioning: Divide the preprocessed data into training, validation, and test sets, with ratios of 70%, 15%, and 15%, respectively. The training set is used for model training, the validation set is used for tuning the model's hyperparameters, and the test set is used for evaluating the model's performance.

[0048] Training the model: Train the model using the training set. During training, adjust the model's parameters to better understand and process text information from various departments. Appropriate hyperparameters such as learning rate, batch size, and number of training epochs can be set. For example, a learning rate of 0.0001, a batch size of 32, and 10 training epochs can be used.

[0049] (4) Model evaluation and optimization Evaluation Metric Selection: Choose appropriate evaluation metrics to assess model performance, such as accuracy, recall, and F1 score. Accuracy measures the proportion of correct predictions made by the model; recall measures the proportion of positive cases correctly identified by the model; and the F1 score is the harmonic mean of accuracy and recall.

[0050] Model optimization: Optimize the model based on the evaluation results. If the model performs poorly in text processing in certain departments, the amount of data for those departments can be increased for retraining; or the model's hyperparameters can be adjusted, such as increasing the number of training epochs or adjusting the learning rate.

[0051] (5) Model application Text understanding: By inputting text such as the patient's medical history and examination reports into the model, the model can understand the professional terminology, chief complaint, and result descriptions. For example, for the statement "I have been experiencing palpitations recently, and my electrocardiogram shows ST-T segment changes," the model can identify "palpitations" as the chief complaint, "ST-T segment changes" as the electrocardiogram result, and analyze possible health problems.

[0052] Information extraction: Extracting key information from text, such as examination items and result values. For example, from "Routine blood test: White blood cell count 6.5 × 10⁻⁶", extracting key information such as examination items and result values. 9 The results from the "complete blood count" and "white blood cell count" were extracted from the "red blood cell count 4.5 × 10¹² / L" result. 9 Information such as "red blood cell count 4.5×10¹² / L" and "red blood cell count 4.5×10¹² / L" are provided.

[0053] Assisted Diagnosis: Based on the model's understanding of the text and information extraction, it provides doctors with auxiliary diagnostic suggestions. For example, for "dizziness, fatigue, hemoglobin concentration 80g / L", the model can suggest the possible presence of anemia and provide further examination suggestions, such as serum iron and ferritin tests.

[0054] The model is applied as follows: (1) Information extraction Identify key entities in the text, including examination item names (such as "vision", "blood pressure", "lung auscultation"), result values ​​(such as "1.0", "120 / 80 mmHg", "clear sound"), descriptive text (such as "pharyngeal congestion", "regular heart rhythm"), chief complaint, and fragments of past medical history.

[0055] Identify semantic relationships: associate values ​​with corresponding items and descriptions with corresponding body parts or symptoms.

[0056] Distinguish between declarative sentences (results / descriptions) and interrogative sentences (doctor's questions).

[0057] Intent recognition: Determine whether the doctor's current speech is describing an examination result, asking about medical history, giving instructions, or chatting with the examinee (which needs to be filtered).

[0058] (2) Support for dialect / accent adaptation: The model is optimized according to the dialect characteristics of the age group of the physical examination population, so that it can better recognize and understand dialects and accents.

[0059] In an internal medicine clinic, the doctor says, "Blood pressure 130 / 80 mmHg, heart rate 75 bpm, clear sounds on auscultation of the heart and lungs." A domain-specific NLP model processes this text, with its information extraction function identifying the key entities "blood pressure," "heart rate," and "auscultation of the heart and lungs," as well as the result values ​​"130 / 80 mmHg," "75 bpm," and "clear sounds," and associating these values ​​with the corresponding items. The intent recognition function determines that the doctor's speech is describing the examination results. If the examinee is elderly and has a regional accent, the system can also accurately recognize their expression through its dialect / accent adaptation function.

[0060] This module can convert speech into accurate, structured text data, improving the quality and efficiency of medical record generation. It also supports dialect / accent adaptation, expanding the system's applicability.

[0061] 4. Structured Electronic Medical Record Template Library Each department has a pre-set, highly structured electronic medical record entry template. These templates include fixed fields, general departmental examination items, departmental specific examination items, standardized options, and free text input fields, as well as the logical relationships between these fields. These templates provide a unified format and standard for subsequent medical record generation.

[0062] Predefined electronic medical record entry templates are provided for different departments, including internal medicine, surgery, ophthalmology, otolaryngology, dentistry, and gynecology. Fixed fields in the templates (such as name, gender, and age) are typically automatically retrieved from the system. General and department-specific examination items are set according to departmental characteristics. Standardized options are presented in dropdown menus, radio buttons / checkboxes, etc., while also allowing free text input. Logical relationships are established between fields; for example, selecting "History of Hypertension" triggers the "Medication Status" field.

[0063] Taking ophthalmology as an example, the electronic medical record entry template includes fixed fields such as "Name," "Gender," and "Age," general departmental examination items such as "Visual Acuity" and "Intraocular Pressure," and department-specific examination items such as "Uncorrected Visual Acuity," "Corrected Visual Acuity," "Slit-Lamp Examination," and "Fundus Examination." Standardized options such as "Visual Acuity Test Results" use a drop-down menu, allowing doctors to select "Normal," "Myopia," or "Hyperopia." If a doctor selects "History of Hypertension," the system automatically pops up the "Medication Information" field for the doctor to fill in.

[0064] The structured electronic medical record template library provides a unified standard and format for medical record entry in various departments, improving the standardization and readability of medical records, while facilitating subsequent medical record management and data statistical analysis.

[0065] 5. Structured template library and auto-fill module Based on the department information provided by the environmental perception module, the corresponding structured template is automatically loaded, and the entity, value, description and other information extracted by the NLP module are intelligently mapped to the corresponding fields of the template to realize the automatic filling of medical records.

[0066] (1) Template matching: Based on the department information determined by the environment perception module, the corresponding template is automatically loaded from the structured electronic medical record template library.

[0067] (2) Field mapping Exact match: When the identified project name matches a template field name, it is directly mapped to the corresponding field in the template. For example, "blood pressure" is mapped to the "blood pressure" field in the template.

[0068] Semantic matching: Mapping identified descriptive content to free text fields. For example, “mild pharyngeal congestion, grade I tonsillar enlargement” is mapped to the “pharynx” description field in an ENT examination.

[0069] Contextualization: Determine the mapping location by combining the previous instruction or the currently focused inspection item.

[0070] Autofill: Fills in numerical values ​​(e.g., 120 / 80) into the corresponding numerical field (e.g., blood pressure), matches standardized descriptions (e.g., "clear voice") to options or fills in the description box, and fills in free text descriptions into the corresponding text boxes.

[0071] In an ophthalmology clinic, the environmental perception module identifies the department as "ophthalmology," and the system automatically loads a structured template for ophthalmology. The NLP module extracts information such as "uncorrected visual acuity 1.0, corrected visual acuity 1.2, intraocular pressure 15 mmHg." The precise matching function maps "uncorrected visual acuity," "corrected visual acuity," and "intraocular pressure" to the corresponding fields in the template, and the autofill function fills in "1.0," "1.2," and "15 mmHg" into the corresponding numerical fields.

[0072] This module enables automatic filling of medical records, reducing the workload of doctors' manual data entry and improving the efficiency and accuracy of medical record generation.

[0073] 6. Human-computer interaction and verification module A preliminary draft of the medical record is generated during or shortly after consultation in the doctor's office for review and revision. It is deeply integrated with the hospital's information system, enabling one-click or automatic data import. Doctors can also make corrections via voice commands or simple touch controls, and voice prompts or pop-up summaries are provided at key confirmation points for doctor confirmation.

[0074] Real-time processing: A preliminary draft of the medical record is generated immediately during or after consultation in the clinic, and doctors can review and modify the draft in a short time.

[0075] Deep integration with hospital information systems: Structured data can be imported into medical record templates in the HIS system with one click or automatically via an interface, avoiding doctors having to perform secondary data entry.

[0076] Voice / Touch Correction: Doctors can correct medical records using voice commands (such as "correct blood pressure to 130 / 85") or simple touch operations (clicking on the incorrect field to modify).

[0077] Key confirmation points: When a major examination item is completed or the examination in the examination room is about to end, the system reminds the doctor to confirm via voice prompt or pop-up summary.

[0078] For example, in a consultation room, after the doctor finishes communicating with the examinee, the system immediately generates a preliminary draft of the medical record. If the doctor finds an error in the "blood pressure" field, they can modify the medical record using the voice command "correct blood pressure to 130 / 85". After all the examination items are completed, the system displays a summary for the doctor to confirm. Once the doctor confirms that everything is correct, the system automatically imports the structured data into the medical record template in the HIS system.

[0079] This module improves the efficiency of doctor-system interaction, making it easier for doctors to review and correct medical records. At the same time, its integration with the hospital information system avoids secondary data entry and improves the accuracy and consistency of data.

[0080] 7. Data Storage and Interface Module The generated electronic medical records are stored in the hospital system, and interfaces with systems such as HIS / LIS are provided to enable data sharing and interaction. Structured electronic medical record data is stored in the hospital system's database through these interfaces, while also supporting data integration with HIS / LIS systems for real-time data synchronization and sharing. The structured electronic medical record data generated by the system is stored in the hospital system's database via these interfaces. Simultaneously, the HIS system can access this data in real time through these interfaces for patient medical record management and medical decision-making. This module achieves secure storage and effective sharing of electronic medical record data, improving the overall efficiency of the hospital information system and facilitating the management and utilization of medical data.

[0081] like Figure 3As shown, on the other hand, an application method for an environment-aware multimodal speech recognition and structured electronic medical record generation system is provided, the method comprising: Real-time sensing and determination of the examinee's ID and current department / clinic; determining the patient's department based on the sensing information and calling the corresponding department's structured template; Collect and preprocess speech within the clinic; The speech is converted into structured information that matches the department's professional terminology and mapped and filled into the corresponding structured template, generating the corresponding medical record draft and sending it to the human-computer interaction and verification module; Doctors review and correct medical records through the human-computer interaction and verification module. Once the review is complete, the medical records are stored in the HIS.

[0082] Please refer to the preceding system description for understanding the above methods and steps.

[0083] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, optionally, electronic device 410 may include a first processor 2001.

[0084] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0085] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0086] The following is combined Figure 4 A detailed description is given of each component of the electronic device 410, for use in... The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as for one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0087] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0088] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0089] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0090] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above system embodiment, and will not be repeated here.

[0091] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0092] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0093] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4(Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0094] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0095] It should be noted that, Figure 4 The structure of the electronic device 410 shown does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] Furthermore, the technical effects of the electronic device 410 can be referenced from the technical effects of the environmentally aware multimodal speech recognition and structured electronic medical record generation system described in the above system embodiments, and will not be repeated here.

[0097] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0098] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0099] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0100] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it may also indicate an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0101] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can be used for a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0102] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different systems to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing system embodiments, and will not be repeated here.

[0105] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0106] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0108] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the system described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A system for generating structured electronic medical records based on environmentally perceptive multimodal speech recognition, characterized in that: The system includes: The environmental perception module is used to perceive and determine the examinee's ID and current department / clinic in real time; A multimodal speech acquisition and preprocessing module is used to acquire and preprocess speech within the clinic. The Natural Language Understanding and Structured Processing module is used to convert speech into structured information that is adapted to the department's professional terminology; A structured electronic medical record template library is used to store predefined structured templates for each department; The structured template matching and autofill module is used to map and fill structured information into the corresponding structured template; The human-computer interaction and verification module is used by doctors to review and correct medical records; The data storage and interface module is used for storing medical records and connecting them with the HIS (Hospital Information System).

2. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The environment sensing module includes: The clinic-department identification unit is used to match the clinic number through a preset "clinic-department" mapping database, or to double-confirm the department corresponding to the current clinic by combining the department code logged in by the doctor; The examinee location and identification unit is used to determine the examinee's current examination room through RFID / Bluetooth beacons or computer vision technology.

3. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The multimodal speech acquisition and preprocessing module includes: Microphone arrays are used to capture the voices of doctors and patients undergoing physical examinations and to distinguish the direction of the speaker; A multimodal speech recognition engine is used to include a high-performance automatic speech recognition (ASR) sub-engine (which converts speech streams to text in real time), a speaker separation and role recognition sub-engine (which distinguishes doctor / examinee speech based on voiceprint, location or preset role), a context-aware sub-engine (which optimizes the recognition of contextual terms by combining departmental environment, examination equipment or template fields) and / or an instruction recognition sub-engine (which recognizes doctor instructions).

4. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The natural language understanding and structuring module includes: The NLP model operation module is used to train and optimize NLP models based on the professional terminology, common complaints, examination items and result descriptions of various departments in the health checkup center. The information extraction submodule is used to call NLP models to identify key entities, semantic relationships, and distinguish between declarative and interrogative sentences. The intent recognition submodule is used to determine the doctor's speech intent; The dialect / accent adaptation submodule is used to adapt to the dialects of the age groups of people undergoing physical examinations.

5. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The structured template matching and auto-fill module includes: The template matching unit is used to automatically load the structured template of the corresponding department based on the department information provided by the environmental perception module. Field mapping unit, used to map structured information to corresponding fields in the template through exact matching, semantic matching or contextual association; The autofill cell is used to fill numerical values ​​into corresponding numerical fields, match standardized descriptions to options or fill them into description boxes, or fill free text into corresponding text boxes.

6. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The human-computer interaction and verification module includes: The real-time processing module is used to generate preliminary draft medical records during or shortly after consultations. The voice / touch correction module allows doctors to modify draft medical records via voice commands or touch operations; The key confirmation point module is used to provide voice prompts or pop-up summaries for doctors to confirm when completing an examination item or ending an examination in the clinic. The HIS integration module is used to import structured data into the HIS system's medical record templates with one click or automatically.

7. The environmentally-aware multimodal speech recognition and structured electronic medical record generation system according to claim 1, characterized in that, The medical record data storage and HIS interface includes: The confirmed electronic medical records will be stored in the hospital system; It supports integration with systems such as HIS / LIS to enable data exchange.

8. An application method of the environmentally aware multimodal speech recognition and structured electronic medical record generation system according to any one of claims 1-7, characterized in that, The method includes: Real-time sensing and determination of the examinee's ID and current department / clinic; determining the patient's department based on the sensing information and calling the corresponding department's structured template; Collect and preprocess speech within the clinic; The speech is converted into structured information that matches the department's professional terminology and mapped and filled into the corresponding structured template, generating the corresponding medical record draft and sending it to the human-computer interaction and verification module; Doctors review and correct medical records through the human-computer interaction and verification module. Once the review is complete, the medical records are stored in the HIS.

9. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in claim 8.

Images