Method, device, equipment and product for automatically transcribing voice into HIS (hospital information system) of outpatient medical record

Abstract

The invention discloses a method, a device, equipment and a product for automatically transcribing an HIS (Hospital Information System) through voice of an outpatient medical record, and relates to the technical field of automatic data entry. The method comprises the following steps: firstly, converting doctor inquiry voice into a text by applying a medical ASR technology, then carrying out entity recognition, term standardization and relation extraction through a medical NLP technology, generating structured JSON data with medical record field names as keys and entity information as values, and then fusing an Accession API and computer vision template matching technology to obtain a text database; according to the method, all the control capable of being filled in the target HIS interface is automatically detected, the mapping relation between the control and the medical record field name is established, finally, the corresponding entity information is extracted from the JSON data according to the mapping relation, and the corresponding control is automatically filled through the RPA or Computer Use technology, so that the medical term recognition accuracy can be remarkably improved, and the medical term recognition efficiency is improved. And seamless integration with a multi-brand HIS system is realized, full-automatic and cross-platform input of voice medical records is achieved, and practical application and popularization are facilitated.

Description

Technical Field

[0001] This invention belongs to the field of automated data entry technology, specifically relating to a method, apparatus, equipment, and product for automatic transcription of outpatient medical records into HIS (Hospital Information System). Background Technology

[0002] In the current outpatient medical process, doctors need to simultaneously examine patients and write electronic medical records. This process takes an average of 3 to 5 minutes per patient, accounting for 25% to 30% of the overall consultation time, and has become one of the main bottlenecks restricting the improvement of outpatient efficiency. To reduce the paperwork burden on doctors, speech-to-text tools have been tried in clinical scenarios. However, general-purpose automatic speech recognition (ASR) systems perform poorly in the highly specialized field of medicine. Their error rate in recognizing professional terms such as drug names, symptoms, and signs remains high (usually exceeding 15%), resulting in low reliability of the transcribed text. Doctors still need to spend a lot of time proofreading and revising.

[0003] More importantly, existing speech-to-text solutions only address the "speech-to-text" conversion problem, failing to achieve automatic "text-to-HIS (Hospital Information System)" input. The transcribed text still requires doctors to manually copy and paste it into the corresponding fields in the HIS system, increasing the number of steps rather than reducing operational complexity, thus failing to fundamentally improve efficiency. For example, some have proposed a voice input method for electronic medical records, but its core function is merely to convert speech to text; subsequent manual input into the HIS is still required, failing to address the core issues of structured recognition of medical entities and automated field filling. Furthermore, while some leading HIS vendors offer built-in voice medical record plugins that enable in-system input, their technology heavily relies on the proprietary data structures and interfaces of specific vendors' HIS systems, making them incompatible with other brands' HIS systems. This leaves hospitals helpless when facing multiple HIS systems coexisting or system upgrades.

[0004] In summary, the existing technology has the following three main drawbacks: (1) Inaccurate identification of medical entities: The general ASR model lacks medical domain knowledge and misidentifies professional terms such as "hemoptysis", "rales" and "metoprolol" as "bleeding", "rales" and "metoprolol", resulting in serious errors in the medical record content; (2) Poor dialect adaptability: In areas where Cantonese, Minnan and other dialects are commonly used for consultation, the recognition rate of the general ASR model is less than 70%, which seriously restricts its clinical application in dialect areas; (3) Cannot be seamlessly integrated with multi-brand HIS: There are huge differences in field naming and interface layout between different HIS vendors. There is a lack of a universal, non-intrusive automatic filling mechanism, which makes manual mapping time-consuming and labor-intensive, and the system deployment and maintenance costs are extremely high.

[0005] Therefore, how to provide a new solution for an intelligent outpatient medical record creation process that aims to achieve "generation while speaking and entry upon generation" is a topic that urgently needs to be studied by those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a method, apparatus, computer equipment, computer-readable storage product, and computer program product for automatic transcription of outpatient medical records into HIS, in order to solve the problems of inaccurate medical entity recognition, poor dialect adaptability, and / or inability to seamlessly integrate with multiple HIS brands in existing speech transcription solutions.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for automatically transcribing outpatient medical records into HIS (Hospital Information System) is provided, including: Acquire doctor's voice data collected by the microphone; The doctor's consultation voice data is processed by using automatic speech recognition technology in the medical field to convert the voice data into text, resulting in doctor's consultation text data. The medical field natural language processing technology is applied to the doctor's consultation text data to perform medical entity recognition, medical terminology standardization and entity relation extraction to obtain a multi-entity relation group, and a structured JSON data containing medical record field names as keys and entity information as keys is generated based on the multi-entity relation group. Automatically discover all fillable controls on the target HIS interface by using the Accessibility API and computer vision template matching. For each control in all the fillable controls, determine the mapping relationship between the corresponding control and the medical record field name; For each control, based on any medical record field name that has the mapping relationship with the corresponding control, extract any entity information corresponding to any medical record field name from the structured JSON data, and apply Robotic Process Automation (RPA) or Computer Use (CPU) technology to automatically fill the entity information into the corresponding control.

[0008] Based on the above-mentioned invention, a new solution is provided that can automatically convert doctors' consultation speech into text and automatically input it into any HIS system. First, medical automatic speech recognition technology is applied to process the doctors' consultation speech data into text data. Then, medical natural language processing technology is applied to the text data for medical entity recognition, medical terminology standardization, and entity relation extraction, resulting in a multi-entity relation group. This generates structured JSON data containing key names of medical record field names and key values ​​of entity information. Then, through the Accessibility API and computer vision template matching, all fillable controls on the target HIS interface are automatically discovered. For each control, the mapping relationship between the corresponding control and the medical record field name is determined. Based on any medical record field name that has a mapping relationship with the corresponding control, any entity information corresponding to that medical record field name is extracted from the structured JSON data. Finally, Robotic Process Automation (RPA) or Computer Applications (CPA) technology is applied. Use this feature to automatically fill in the information of any entity into the corresponding control. This not only improves the accuracy of medical entity recognition, but also allows for seamless integration with multiple HIS brands to achieve the goal of automatically transcribing doctors' consultation voice into any HIS, facilitating practical application and promotion.

[0009] In one possible design, medical-grade automatic speech recognition technology is applied to process the doctor's consultation speech data into text, resulting in doctor's consultation text data, including: The doctor's consultation voice data was processed for dialect classification to obtain dialect classification results; Based on the dialect classification results, the speech-to-text processing path is selected as follows: if the dialect classification result is standard Mandarin, the pre-trained general acoustic model is directly used to process the doctor's consultation speech data into speech-to-text; if the dialect classification result is a certain dialect, the accent adapter network corresponding to that certain dialect is activated in the multi-accent adapter network, and it works in conjunction with the general acoustic model to process the doctor's consultation speech data into speech-to-text. The general acoustic model uses a Conformer encoder and a hybrid decoding structure consisting of a CTC decoder and an attention mechanism. During the decoding process of the speech-to-text processing, a medical hot word library is dynamically injected through a finite state converter to enhance the recognition capability of medical terms, resulting in the final doctor's consultation text data.

[0010] In one possible design, natural language processing techniques from the medical field are applied to perform medical entity recognition, medical terminology standardization, and entity relation extraction on the doctor's consultation text data, including: The BioBERT and GlobalPointer models were used to perform medical entity recognition processing on the doctor's consultation text data, and multiple types of medical entities were identified, including symptoms, diseases, drugs, examination item names, dosages and / or frequencies. Based on a medical knowledge graph, all identified medical entities are standardized with medical terminology to map colloquial medical entities into new medical entities with standardized terminology. The medical knowledge graph is constructed based on the Unified Medical Language System (UMLS) and a self-built drug database. Entity relation extraction is performed on all new medical entities to construct a ternary entity relation group with a "symptom-disease-drug" association.

[0011] In one possible design, for each control among all the populateable controls, the mapping relationship between the corresponding control and the medical record field name is determined, including: For each control in all the fillable controls, the semantic similarity between each medical record field name and the corresponding control label is calculated using the Sentence-BERT model. The initial mapping result is obtained by threshold comparison. The initial mapping result is then confirmed and corrected by regular expressions and / or a preset dictionary to obtain the final mapping result. The final mapping result contains a medical record field name that has a mapping relationship with the corresponding control.

[0012] In one possible design, the method further includes: The system monitors the operation status in real time during the automatic data entry process. When a data entry failure event is detected due to network latency, pop-up window obstruction, or unavailable controls, the system automatically triggers an abnormal rollback strategy. The abnormal rollback strategy refers to automatically taking a screenshot to save the current state, recording detailed error logs, rolling back the entered data, and automatically retrying when conditions permit, with a maximum of 3 retries. And / or, the entire process record includes an audit log containing the doctor's consultation voice data, the doctor's consultation text data, the mapping relationship, the final filling result, and the operation timestamp, and the audit log is stored in a three-dimensional encrypted manner according to the patient, doctor, and timestamp to meet the traceability requirements of the electronic medical record management standard.

[0013] In one possible design, the method further includes: Automatically capture any modifications made by doctors to the doctor's consultation text data and generate a differential text dataset; Using the aforementioned differential text dataset, the models in the automatic speech recognition technology and the natural language processing technology in the medical field are incrementally fine-tuned using the low-rank adaptive technique LoRA for large language models. The models in the automatic speech recognition technology and natural language processing technology in the medical field are updated weekly to achieve continuous model evolution and make the update process unnecessary for downtime and retraining.

[0014] Secondly, an apparatus for automatically transcribing outpatient medical records into HIS is provided, including a voice data acquisition unit, a voice-to-text conversion unit, a natural language processing unit, a fill control discovery unit, a mapping relationship determination unit, and an information automatic filling unit. The voice data acquisition unit is used to acquire doctor's consultation voice data collected by the microphone; The speech-to-text conversion unit is communicatively connected to the speech data acquisition unit and is used to apply medical automatic speech recognition technology to process the doctor's consultation speech data into speech-to-text data to obtain the doctor's consultation text data. The natural language processing unit is communicatively connected to the speech-to-text conversion unit. It is used to apply medical natural language processing technology to perform medical entity recognition, medical terminology standardization, and entity relationship extraction on the doctor's consultation text data to obtain a multi-entity relationship group. Based on the multi-entity relationship group, it generates structured JSON data containing key names of medical record field names and key values ​​of entity information. The fillable control discovery unit is used to automatically discover all fillable controls on the target HIS interface through the Accessibility API and computer vision template matching. The mapping relationship determination unit is communicatively connected to the natural language processing unit and the fill control discovery unit, respectively, and is used to determine the mapping relationship between the corresponding control and the medical record field name for each control in all fillable controls; The information autofill unit is communicatively connected to the natural language processing unit and the mapping relationship determination unit, respectively. It is used to extract any entity information corresponding to any medical record field name that has the mapping relationship with the corresponding control from the structured JSON data, and automatically fill the entity information into the corresponding control by applying Robotic Process Automation (RPA) or Computer Use (CUSE) technology.

[0015] Thirdly, the present invention provides a computer device comprising a storage module, a processing module, and a transceiver module connected in sequence for communication, wherein the storage module is used to store a computer program, the transceiver module is used to send and receive messages, and the processing module is used to read the computer program and execute the method as described in the first aspect or any possible design in the first aspect.

[0016] Fourthly, the present invention provides a computer-readable storage product having instructions stored thereon that, when executed on a computer, perform the method described in the first aspect or any possible design in the first aspect.

[0017] Fifthly, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implement the method as described in the first aspect or any possible design in the first aspect.

[0018] The beneficial effects of the above scheme are: (1) This invention creatively provides a new solution that can automatically convert doctors' consultation voice into text and automatically input it into any HIS system. First, it applies medical automatic speech recognition technology to process the doctors' consultation voice data into text, obtaining doctors' consultation text data. Then, it applies medical natural language processing technology to process the text data into medical entity recognition, medical terminology standardization, and entity relationship extraction, obtaining a multi-entity relationship group and generating structured JSON data containing medical record field names as keys and entity information as keys. Then, it automatically discovers all fillable controls on the target HIS interface through the Accessibility API and computer vision template matching method, and determines the mapping relationship between the corresponding control and the medical record field name for each control. Based on any medical record field name that has a mapping relationship with the corresponding control, it extracts any entity information corresponding to any medical record field name from the structured JSON data. Finally, it applies Robotic Process Automation (RPA) technology or Computer Applications (Computer) technology. Use this function to automatically fill the information of any entity into the corresponding control. This not only improves the accuracy of medical entity recognition, but also allows for seamless integration with multiple HIS systems to achieve the goal of automatically transcribing doctors' consultation voice into any HIS system. (2) It can achieve full-process automation and efficiency leap, which can shorten the manual writing and input process that takes 3-5 minutes to be completed automatically in about 20 seconds, improving outpatient efficiency by about 30%, and fundamentally freeing doctors from paperwork burden; (3) It can significantly improve the recognition accuracy. Specifically, by using the "general acoustic model + dialect adapter" and "medical hot words FST dynamic injection", the recognition error rate of dialects such as Cantonese and Minnan can be reduced from 28% to below 9%, and the recognition accuracy of medical terms can be increased to over 95%. Furthermore, by using the domain-optimized BioBERT and GlobalPointer models, it can achieve accurate recognition and standardization of 42 types of medical entities (F1=94.2%), reducing the terminology error rate in medical records by approximately 80%. (4) It has excellent cross-system compatibility and deployment convenience. It integrates the hybrid perception technology of Accessibility API and CV template matching, and combines intelligent semantic mapping based on Sentence-BERT to achieve non-intrusive and adaptive discovery and matching of more than 10 different brands of HIS interface controls; and the mapping results can be cached and reused, which shortens the deployment cycle of cross-HIS systems from days to hours, significantly reducing the cost of hospital information transformation. (5) It can improve robustness, compliance and self-evolution capability. That is, by combining RPA with the abnormal rollback strategy, the success rate of data entry is guaranteed (>99%), which meets the requirements of high reliability in clinical operation. It also has a complete three-dimensional audit log of the whole process, which fully meets the traceability requirements of the "Electronic Medical Record Management Standard". Furthermore, based on the LoRA online incremental learning closed loop, the whole solution can continuously optimize itself from the doctor's modification, and update weekly without being noticed, which ensures the continuous evolution of performance and long-term applicability, and is convenient for practical application and promotion. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the method for automatically transcribing outpatient medical records into HIS (Hospital Information System) as provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the device for automatically transcribing outpatient medical records into HIS, as provided in an embodiment of this application.

[0022] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these embodiments without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0024] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0025] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0026] Example like Figure 1 As shown, the method for automatically transcribing outpatient medical records into HIS provided in the first aspect of this embodiment can be executed, but is not limited to, by computer devices with certain computing resources, such as servers, personal computers (PCs, referring to a type of multi-purpose computer suitable for personal use in terms of size, price, and performance; desktops, laptops, mini-laptops, tablets, and ultrabooks are all considered personal computers), smartphones, personal digital assistants (PDAs), or wearable devices. Figure 1 As shown, the method for automatically transcribing outpatient medical records into HIS includes, but is not limited to, the following steps S1 to S6.

[0027] S1. Acquire doctor's consultation voice data collected by the microphone.

[0028] In step S1, the doctor's consultation voice data refers to the voice data collected during the consultation process. Specifically, the microphone can be a commercially available third-party microphone (beamforming directional pickup, SNR≥30dB), or it can be connected to a mobile phone microphone (automatic gain control).

[0029] S2. Apply automatic speech recognition technology in the medical field to process the doctor's consultation voice data into text, and obtain the doctor's consultation text data.

[0030] In step S2, in order to achieve the purpose of joint recognition of medical dialects and terminology, preferably, the doctor's consultation voice data is processed by speech-to-text using medical automatic speech recognition technology to obtain doctor's consultation text data, including but not limited to the following steps S21 to S23.

[0031] S21. Perform dialect classification processing on the doctor's consultation voice data to obtain the dialect classification result.

[0032] In step S21, the doctor's consultation voice data can be imported into a pre-trained dialect classification model to obtain the dialect classification result: standard Mandarin, Cantonese, Minnan, or other dialects.

[0033] S22. Based on the dialect classification result, the speech-to-text processing path is selected as follows: If the dialect classification result is standard Mandarin, the pre-trained general acoustic model is directly used to process the doctor's consultation speech data into speech-to-text. If the dialect classification result is a certain dialect, the accent adapter network corresponding to that dialect is activated in the multi-accent adapter network, and it works in conjunction with the general acoustic model to process the doctor's consultation speech data into speech-to-text. The general acoustic model uses a Conformer encoder and a hybrid decoding structure consisting of a CTC decoder and an attention mechanism.

[0034] In step S22, the multi-accent adapter network is a parameter-efficient transfer learning technique in speech recognition systems. It enables the model to quickly adapt to multiple dialects or accents by embedding a lightweight adapter module into a pre-trained general speech recognition model without retraining the entire large model. Thus, by activating the accent adapter network corresponding to a certain dialect, the word recognition error rate of dialects such as Cantonese or Minnan in medical scenarios can be significantly reduced from 28% to below 9%. The Conformer encoder, the CTC (Connectionist Temporal Classification) decoder, and the Attention mechanism are all existing encoding / decoding technologies. The Conformer encoder is used to capture subtle local features (such as consonant plosives) and global contextual dependencies (such as intonation and rhythm) of the speech signal in parallel, significantly improving the modeling ability for complex medical pronunciations. The CTC decoder introduces special "whitespace" characters to solve the problem of misalignment between the number of speech frames and the length of the output text, and efficiently compresses repetitive pronunciations and silent segments through dynamic programming, achieving efficient and stable frame-level alignment and ensuring transcription speed. The Attention mechanism simulates the "attention" of humans during dictation, dynamically assigning weights to different parts of the input speech, focusing on the most relevant contextual information. This enhances the model's understanding of long-distance context and effectively corrects ambiguities caused by dialects or colloquial expressions. Specifically, the general acoustic model has as many as 85M (M means millions) parameters and can be trained based on the following training corpus: 3,000 hours of real outpatient recordings (18 departments and 7 dialects) and 500,000 medical TTS (Text-To-Speech) data.

[0035] S23. In the decoding process of the speech-to-text processing, a medical hot word library is dynamically injected through a finite state converter to enhance the recognition ability of medical terms, and the final doctor's consultation text data is obtained.

[0036] In step S23, the medical hot word database specifically contains at least 300,000 medical hot words such as drug names, disease names and / or examination item names, and injects them into the decoding path in real time through FST (Finite-State Transducer) to ensure that the accuracy of medical term recognition is ≥95%.

[0037] S3. Apply natural language processing technology in the medical field to perform medical entity recognition, medical terminology standardization and entity relation extraction on the doctor's consultation text data to obtain a multi-entity relation group, and generate structured JSON data containing key names of medical record field names and key values ​​of entity information based on the multi-entity relation group.

[0038] In step S3, specifically, medical natural language processing technology is applied to the doctor's consultation text data to perform medical entity recognition, medical terminology standardization, and entity relationship extraction processing, including but not limited to the following steps S31 to S33.

[0039] S31. Apply the BioBERT model and GlobalPointer model to perform medical entity recognition processing on the doctor's consultation text data to identify multiple types of medical entities, including but not limited to symptoms, diseases, drugs, examination item names, dosages and / or frequencies.

[0040] In step S31, the BioBERT model is a pre-trained language model optimized for the biomedical field. Based on BERT (Bidirectional Encoder Representations from Transformers), it uses a vast amount of medical literature (such as PubMed abstracts) for domain-adaptive pre-training, thereby deeply integrating medical knowledge and more accurately understanding the contextual semantics of professional terms such as "hemoptysis" and "rales," providing a strong semantic foundation for medical entity recognition. The GlobalPointer model is an efficient decoding mechanism for named entity recognition. Through global normalization, it treats entity recognition as a set prediction problem, capable of decoding all possible entity spans (including nested entities) in the text at once, effectively solving the difficulties of traditional methods in handling nested medical entities such as "acute severe pancreatitis" (which may contain nested "severe pancreatitis"). Therefore, by combining the BioBERT model and the GlobalPointer model, the F1 score for medical entity recognition processing can reach 94.2%.

[0041] S32. Based on the medical knowledge graph, all identified medical entities are standardized with medical terminology to map colloquially expressed medical entities into new medical entities with standardized terminology, wherein the medical knowledge graph is constructed based on the Unified Medical Language System (UMLS) and a self-built drug database.

[0042] In step S32, for example, "metoprolol" can be standardized as "metoprolol".

[0043] S33. Extract entity relationships from all new medical entities and construct a ternary entity relationship group with a "symptom-disease-drug" association.

[0044] S4. Automatically discover all fillable controls on the target HIS interface through the Accessibility API and computer vision template matching.

[0045] In step S4, the Accessibility API is used to programmatically read the UI (User Interface) control tree of the target HIS application. This process can obtain the precise type, programmatic name, screen coordinates, and hierarchical relationship of all standard controls (such as text boxes and drop-down lists) in milliseconds. For custom controls that cannot be fully identified or labeled by the API, the computer vision template matching method can be activated to take a screenshot of the current screen and perform multi-scale matching and OCR (Optical Character Recognition) with a preset control visual feature library (such as input box borders, button icons, or label text templates) to locate and supplement non-standard or graphical fillable areas. Finally, the results of the two types are combined to generate a complete list of fillable controls containing control semantic information and precise coordinates (i.e., all fillable controls on the target HIS interface).

[0046] S5. For each control in all the fillable controls, determine the mapping relationship between the corresponding control and the medical record field name.

[0047] In step S5, specifically, for each control among all the fillable controls, the mapping relationship between the corresponding control and the medical record field name is determined. This includes, but is not limited to: for each control among all the fillable controls, using the Sentence-BERT model to calculate the semantic similarity between each medical record field name and the corresponding control label, obtaining the corresponding initial mapping result through threshold comparison, and further confirming and correcting the initial mapping result through regular expressions and / or a preset dictionary to obtain the corresponding final mapping result. The final mapping result contains a medical record field name that has a mapping relationship with the corresponding control. The Sentence-BERT model is a pre-trained model that generates semantic vector representations for sentences. It modifies the BERT network structure to efficiently encode a complete sentence into a fixed-dimensional semantic vector, thus obtaining the semantic vectors of the medical record field name and the control label respectively. Then, the semantic similarity between the two is calculated conventionally, and the initial mapping result is obtained conventionally based on the comparison between the calculated result and a preset semantic similarity threshold. Furthermore, the initial final mapping result can be stored encrypted using JSON Schema for subsequent reuse by HIS systems of the same brand.

[0048] S6. For each control, based on any medical record field name that has the mapping relationship with the corresponding control, extract any entity information corresponding to any medical record field name from the structured JSON data, and apply Robotic Process Automation (RPA) or Computer Use (CPU) technology to automatically fill the entity information into the corresponding control.

[0049] In step S6, the Robotic Process Automation (RPA) technology is a technique that uses software robots to automatically execute rule-based, repetitive computer operations. In this embodiment, RPA serves as a high-level process orchestration framework, responsible for scheduling the entire data entry process: reading structured data, locating controls based on mapping relationships, filling in information in logical order, handling exceptions, and submitting forms. It provides reliable process control and error recovery mechanisms. The computer usage technology specifically refers to directly simulating low-level human-computer interaction operations such as keyboard input, mouse clicks, and focus switching through system-level APIs. In this solution, it is the underlying execution unit called by the RPA framework, ensuring that it can simulate real user operations with the highest compatibility without relying on the HIS database interface, achieving non-intrusive control of any HIS interface (i.e., simulating manual keyboard and mouse operations without calling the HIS database).

[0050] In addition, to achieve the purpose of automatic data entry rollback in case of an error, the method preferably includes, but is not limited to: real-time monitoring of the operation status during the automatic data entry process, and automatically triggering an error rollback strategy when a data entry failure event caused by network latency, pop-up window obstruction, or unavailability of controls is detected. The error rollback strategy refers to automatically taking a screenshot to save the current state, recording detailed error logs, rolling back the entered data, and automatically retrying when conditions permit, with the number of retries not exceeding 3.

[0051] In addition, in order to achieve the purpose of automatically recording audit logs, the preferred method also includes, but is not limited to: recording the entire process of audit logs including the doctor's consultation voice data, the doctor's consultation text data, the mapping relationship, the final filling result and operation timestamp, and storing the audit logs in a three-dimensional encrypted manner according to the patient, doctor and timestamp to meet the traceability requirements of electronic medical record management standards.

[0052] Furthermore, to achieve the objectives of auditing and incremental learning, the preferred method also includes, but is not limited to: first, automatically capturing any modifications made by doctors to the doctor's consultation text data to generate a differential text dataset; then, using the differential text dataset, incrementally fine-tuning the models in the automatic speech recognition technology and the natural language processing technology in the medical field through the low-rank adaptation technique LoRA for large language models; updating the models in the automatic speech recognition technology and the natural language processing technology in the medical field on a weekly basis to achieve continuous model evolution and eliminate the need for downtime retraining during the update process. The low-rank adaptation technique LoRA is an efficient fine-tuning technique for large language models. Its core idea is to freeze all parameters of the original large model and only insert a small number of trainable low-rank decomposition matrices for fine-tuning. This method can achieve rapid adaptation of the model to new tasks or domains with extremely low parameter counts (usually less than 1% of the original model), effectively avoiding the huge computational cost and "catastrophic forgetting" risk of retraining the entire model, making it very suitable for practical business scenarios requiring continuous online learning.

[0053] Therefore, based on the method for automatically transcribing outpatient medical record speech to HIS described in steps S1 to S6 above, a new solution is provided that can automatically convert doctors' consultation speech into text and automatically input it into any HIS system. First, medical-domain automatic speech recognition technology is applied to process the doctors' consultation speech data into text, obtaining doctors' consultation text data. Then, medical-domain natural language processing technology is applied to the text data for medical entity recognition, medical terminology standardization, and entity relation extraction, obtaining a multi-dimensional entity relation group. This generates structured JSON data containing medical record field names as keys and entity information as keys. Then, through the Accessibility API and computer vision template matching, all fillable controls on the target HIS interface are automatically discovered. For each control, the mapping relationship between the corresponding control and the medical record field name is determined. Based on any medical record field name that has a mapping relationship with the corresponding control, any entity information corresponding to that medical record field name is extracted from the structured JSON data. Finally, Robotic Process Automation (RPA) technology or Computer Applications (CPA) technology is applied. Use this feature to automatically fill in the information of any entity into the corresponding control. This not only improves the accuracy of medical entity recognition, but also allows for seamless integration with multiple HIS brands to achieve the goal of automatically transcribing doctors' consultation voice into any HIS, facilitating practical application and promotion.

[0054] like Figure 2 As shown, the second aspect of this embodiment provides a virtual device for implementing the method of automatic transcription of outpatient medical record voice into HIS as described in the first aspect, including a voice data acquisition unit, a voice-to-text conversion unit, a natural language processing unit, a fill control discovery unit, a mapping relationship determination unit, and an information automatic filling unit; The voice data acquisition unit is used to acquire doctor's consultation voice data collected by the microphone; The speech-to-text conversion unit is communicatively connected to the speech data acquisition unit and is used to apply medical automatic speech recognition technology to process the doctor's consultation speech data into speech-to-text data to obtain the doctor's consultation text data. The natural language processing unit is communicatively connected to the speech-to-text conversion unit. It is used to apply medical natural language processing technology to perform medical entity recognition, medical terminology standardization, and entity relationship extraction on the doctor's consultation text data to obtain a multi-entity relationship group. Based on the multi-entity relationship group, it generates structured JSON data containing key names of medical record field names and key values ​​of entity information. The fillable control discovery unit is used to automatically discover all fillable controls on the target HIS interface through the Accessibility API and computer vision template matching. The mapping relationship determination unit is communicatively connected to the natural language processing unit and the fill control discovery unit, respectively, and is used to determine the mapping relationship between the corresponding control and the medical record field name for each control in all fillable controls; The information autofill unit is communicatively connected to the natural language processing unit and the mapping relationship determination unit, respectively. It is used to extract any entity information corresponding to any medical record field name that has the mapping relationship with the corresponding control from the structured JSON data, and automatically fill the entity information into the corresponding control by applying Robotic Process Automation (RPA) or Computer Use (CUSE) technology.

[0055] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be found in the method described in the first aspect, and will not be repeated here.

[0056] like Figure 3 As shown, the third aspect of this embodiment provides a computer device for executing the method of automatic transcription of outpatient medical records into HIS as described in the first aspect. The device includes a storage module, a processing module, and a transceiver module connected in sequence. The storage module stores a computer program, the transceiver module sends and receives messages, and the processing module reads the computer program and executes the method described in the first aspect. Specifically, the storage module may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processing module may, but is not limited to, use a microprocessor of the STM32F105 series. Furthermore, the computer device may also include, but is not limited to, a power supply module, a display screen, and other necessary components.

[0057] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the method described in the first aspect, and will not be repeated here.

[0058] This fourth aspect of the embodiment provides a computer-readable storage product that stores instructions comprising the method for automatically transcribing outpatient medical records into a Hospital Information System (HIS) as described in the first aspect. Specifically, the computer-readable storage product stores instructions that, when executed on a computer, perform the method described in the first aspect. The computer-readable storage product refers to a data storage medium and may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0059] The working process, working details and technical effects of the aforementioned computer-readable storage product provided in the fourth aspect of this embodiment can be found in the method described in the first aspect, and will not be repeated here.

[0060] This fifth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for automatic transcription of outpatient medical record voice into a Hospital Information System (HIS) as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0061] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of outpatient medical record voice automatic transcription HIS, characterized by, The method comprises the following steps: acquiring doctor consultation voice data collected by a pickup device; applying automatic speech recognition technology in the medical field to perform speech-to-text processing on the doctor consultation voice data to obtain doctor consultation text data; applying natural language processing technology in the medical field to perform medical entity recognition, medical terminology standardization, and entity relationship extraction processing on the doctor consultation text data to obtain a multi-entity relationship group, and generating structured JSON data containing a key name of a medical record field name and a key value of entity information according to the multi-entity relationship group; automatically discovering all fillable controls on a target HIS interface through an accessibility auxiliary function interface Accessibility API and a computer vision template matching method; determining a mapping relationship between a corresponding control and the medical record field name for each control in the all fillable controls; for each control, extracting any entity information corresponding to any medical record field name having the mapping relationship with the corresponding control from the structured JSON data, and applying robot process automation technology RPA or computer use technology‌Computer Use to automatically fill the any entity information into the corresponding control.

2. The method of claim 1, wherein, applying automatic speech recognition technology in the medical field to perform speech-to-text processing on the doctor consultation voice data to obtain doctor consultation text data, comprising: performing dialect classification processing on the doctor consultation voice data to obtain a dialect classification result; according to the dialect classification result, selecting a speech-to-text processing path in the following manner: if the dialect classification result is standard Mandarin, directly using a pre-trained general acoustic model to perform speech-to-text processing on the doctor consultation voice data, and if the dialect classification result is a certain dialect, activating an accent Adapter network corresponding to the certain dialect in a multi-accent Adapter network, and cooperatively performing speech-to-text processing on the doctor consultation voice data with the general acoustic model, wherein the general acoustic model uses a Conformer encoder and a hybrid decoding structure composed of a CTC decoder and an attention mechanism Attention; in the decoding process of the speech-to-text processing, a medical hot word library is dynamically injected through a finite state transducer to enhance the medical terminology recognition capability to obtain the final doctor consultation text data.

3. The method of claim 1, wherein, applying natural language processing technology in the medical field to perform medical entity recognition, medical terminology standardization, and entity relationship extraction processing on the doctor consultation text data, comprising: applying a BioBERT model and a GlobalPointer model to perform medical entity recognition processing on the doctor consultation text data to recognize a plurality of medical entities, wherein the plurality of medical entities include symptoms, diseases, drugs, examination item names, dosages, and / or frequencies; The identified medical entities are standardized into medical terms based on a medical knowledge graph to map the colloquial medical entities into new medical entities with standard terminology, wherein the medical knowledge graph is constructed based on the Unified Medical Language System (UMLS) and a self-built drug library; All new medical entities are subjected to entity relationship extraction processing to construct a three-entity relationship group in the form of a "symptom-disease-drug" association relationship as a three-entity relationship group.

4. The method of claim 1, wherein, For each control in the entire fillable control, the mapping relationship between the corresponding control and the medical record field name is determined, including: For each control in the entire fillable control, the semantic similarity between each medical record field name and the corresponding control label is calculated using a Sentence-BERT model, and the initial mapping result is obtained through threshold comparison, and the initial mapping result is confirmed and modified again through regular expressions and / or a preset dictionary to obtain the final mapping result, wherein the final mapping result contains a certain medical record field name that has a mapping relationship with the corresponding control.

5. The method of claim 1, wherein, The method further comprises: During the automatic filling process, the operation state is monitored in real time, and when a filling failure event caused by network delay, pop-up window blocking, or control unavailability is detected, an abnormal rollback strategy is automatically triggered, wherein the abnormal rollback strategy refers to automatically taking a screenshot to save the on-site state, recording detailed error logs, rolling back the filled data, and automatically retrying when conditions permit, with the number of retries not exceeding 3 times; And / or, the full-process record includes the doctor's voice data, the doctor's text data, the mapping relationship, the final filling result, and the operation timestamp audit log, and the audit log is stored in three dimensions of patient, doctor, and timestamp to meet the electronic medical record management specification traceability requirements.

6. The method of claim 1, wherein, The method further comprises: Automatically capturing any modifications made by the doctor to the doctor's text data to generate a difference text data set; Using the difference text data set, the models in the medical field automatic speech recognition technology and the medical field natural language processing technology are incrementally fine-tuned through the low-rank adaptive technology (LoRA) of a large language model; The models in the medical field automatic speech recognition technology and the medical field natural language processing technology are updated on a weekly basis to realize continuous evolution of the models and make the updating process unnecessary.

7. An apparatus for outpatient medical record voice automatic transcription HIS, characterized by, It includes a voice data acquisition unit, a voice text conversion unit, a natural language processing unit, a fillable control discovery unit, a mapping relationship determination unit, and an information automatic filling unit. The voice data acquisition unit is configured to acquire doctor's voice data collected by a sound pickup device. The voice text conversion unit is communicatively connected to the voice data acquisition unit and is configured to apply medical field automatic speech recognition technology to perform voice-to-text processing on the doctor's voice data to obtain doctor's text data. The natural language processing unit is communicatively connected to the voice text conversion unit and is configured to apply medical field natural language processing technology to perform natural language processing on the doctor's text data to obtain a mapping relationship between the doctor's text data and a fillable control. The natural language processing unit is in communication connection with the speech text conversion unit, and is configured to apply medical field natural language processing technology to perform medical entity recognition, medical terminology standardization, and entity relationship extraction processing on the doctor inquiry text data, to obtain a multi-entity relationship group, and to generate structured JSON data containing a key name of a medical record field name and a key value of entity information according to the multi-entity relationship group. The filling control discovery unit is configured to automatically discover all fillable controls on the target HIS interface through an accessibility auxiliary function interface Accessibility API and a computer vision template matching manner. The mapping relationship determination unit is in communication connection with the natural language processing unit and the filling control discovery unit, respectively, and is configured to determine, for each control in the all fillable controls, a mapping relationship between the corresponding control and the medical record field name. The information automatic filling unit is in communication connection with the natural language processing unit and the mapping relationship determination unit, respectively, and is configured to, for the each control, extract any entity information corresponding to any medical record field name having the mapping relationship with the corresponding control from the structured JSON data, and apply robot process automation technology RPA or computer use technology‌Computer Use to automatically fill the any entity information into the corresponding control.

8. A computer device, comprising: The computer readable storage product has instructions stored thereon, and when the instructions are executed on a computer, the method of any one of claims 1-6 is executed.

9. A computer readable storage product characterized by The computer readable storage product has instructions stored thereon, and when the instructions are executed on a computer, the method of any one of claims 1-6 is executed.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instructions, when executed on a computer, implement the method of any one of claims 1-6.

Images