A method for automatically generating standardized text of surgical records based on speech recognition and semantic fine-tuning

Abstract

The application belongs to the technical field of operation records, and provides a method for automatically generating standardized texts of operation records based on voice recognition and semantic fine-tuning, which comprises the following steps: voice collection, slice construction, voice recognition, text verification, secondary recognition, secondary verification, modified text receiving, vocabulary elimination, standardized text generation and expert determination; the two models are fine-tuned by LoRA, so that the term recognition and unstructured text processing capability of the models in the ophthalmic professional scene are improved; the slice correlation degree analysis and multi-level verification of the recognized text enhance the text recognition reliability and accuracy of the ASR model fine-tuned by LoRA in the professional scene; the text recognition accuracy is improved through text compensation recognition and external modified text receiving, so that the subsequent steps of suspicious content input are avoided; the AI hallucination is intercepted through the hallucination inhibition mechanism and expert review, so that the standardization and accuracy of the operation records are ensured.

Description

Technical Field

[0001] This invention relates to the field of surgical record technology, and in particular to a method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning. Background Technology

[0002] Automatic speech recognition is abbreviated as ASR; large language models are abbreviated as LLMs. Surgical records are crucial medical documents in ophthalmology clinics. They are not only the core basis for postoperative patient management and follow-up, but also key legal evidence in medical disputes, and carry important data value for medical teaching and research. Although the widespread adoption of electronic medical records has significantly improved the standardization and digitization of surgical records, ophthalmologists still face a heavy paperwork burden. Traditional manual entry, whether by handwriting or keyboard input, is not only time-consuming, but also disrupts clinical workflows. This administrative burden is a significant factor leading to physician burnout, which may potentially affect the quality of patient care. In addition, even though manual recording is considered the gold standard, its accuracy is not impeccable; related studies show that its error rate can be as high as 4%. Core clinical pain points and bottlenecks in medical digital transformation: 1) Heavy paperwork burden leads to burnout: Traditional manual entry or simple template copying recording modes are time-consuming and laborious, seriously squeezing out doctors' time for clinical care and academic research, which is one of the key factors leading to physician burnout, and may even potentially affect the quality of patient care. Complex ophthalmic surgeries and ocular trauma surgeries are highly diverse and involve significant intraoperative uncertainty. These surgeries often involve multiple combined operations on the cornea, lens, vitreous body, and retina. Surgical plans are non-linear and require real-time adjustments based on intraoperative findings, such as intraoperative decisions to perform lens resection, retinal incision, or change the implant. 2) The complex nature of these surgeries renders traditional templates ineffective: This highly unstructured surgical process renders traditional checklist-based or fixed-template electronic medical records completely useless. Doctors must perform extensive manual text entry, which is highly susceptible to overlooking crucial operational details due to fatigue. 3) Manual recording introduces objective errors: Existing research indicates that the error rate of manual recording in high-intensity clinical workflows can be as high as 4%, and it is easily affected by fatigue, leading to logical deviations. This has become a digital bottleneck restricting the improvement of ophthalmology department management efficiency.

[0003] The current state of AI technology applications in the medical field presents several problems: 1) Low accuracy in recognizing medical terminology: ASR (Automatic Speech Recognition) technology converts spoken language into text using computer programs and has been applied in fields such as radiology, pathology, and emergency medicine since the 1980s. However, due to the complexity of natural language and the highly specialized nature of medical terminology, early ASR tools failed to gain widespread application. In recent years, the introduction of deep learning technology has significantly improved the accuracy and effectiveness of ASR. Among them, the Whisper model released by OpenAI has performed exceptionally well in multilingual recognition tasks and has been explored for application in neurosurgery and emergency medicine. However, when faced with ophthalmological terms such as vitrectomy, internal limiting membrane removal, and perfluoropropane gas filling, the recognition accuracy drops significantly, failing to meet clinical precision requirements. 2) Insufficient ability to handle unstructured scenarios: The rapid development of LLMs (Limited Language Management Systems), especially their capabilities in text rewriting and summarization, has provided new solutions for converting spoken text into standardized medical documents. Recently, Xu et al. proposed the LAOS system, exploring the feasibility of generating ophthalmological documents using Paraformer ASR+LLM. However, among all document types, the study found that surgical records performed the worst. Furthermore, when faced with complex, highly unstructured retinal surgeries such as vitrectomy, the model's performance significantly declined compared to more standardized surgeries like cataracts. 3) AI illusions pose medical safety risks: General-purpose models lacking domain knowledge constraints are prone to generating seemingly coherent but medically inaccurate content—a kind of rationalization illusion. Highly fluent generated text may also mislead doctors into overlooking potential logical errors or numerical biases, which is unacceptable in rigorous medical documentation. Simply assessing the similarity of generated text, such as ROUGE and BLEU scores, is insufficient to measure the system's clinical value. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning, which solves the problems of low accuracy in recognizing professional terms, insufficient ability to handle unstructured scenarios, and potential medical safety hazards caused by AI illusions in existing methods.

[0005] To achieve the above objectives, the present invention provides the following solution: A method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning, comprising: The oral recording of the target physician during surgery is collected to obtain the recording to be identified. The recording to be identified is segmented according to the natural speech pauses to obtain the initial speech segments. The spatial correlation degree of adjacent initial speech segments is calculated using the speech segment spatial correlation degree judgment formula to obtain the spatial correlation degree, and the initial speech segments with spatial correlation degree exceeding a set threshold are spliced ​​together to obtain a speech segment set. The speech slice set is used to perform speech recognition using a pre-selected optimal ASR model to obtain the initial recognition text; the optimal ASR model includes: Whisper-V2+LoRA; The pre-trained ophthalmic surgery text judgment model is used to verify the terminology rationality, logical coherence, and numerical validity of each initial identified text, and the text verification results are obtained. The initial recognized text whose text verification result is incorrect is marked as suspicious text, and the speech slice corresponding to the suspicious text and the adjacent speech slices are combined and input into the optimal ASR model for secondary recognition to obtain the secondary recognized text; The secondary recognition text is updated to the initial recognition text to obtain the verification optimized text. The verification optimized text is then verified using the ophthalmic surgery text judgment model to obtain the secondary verification result. If the secondary verification result meets the preset requirements, the verification optimized text is set as the basic colloquial text. When the secondary verification result does not meet the preset requirements, the text corresponding to the non-compliance in the secondary verification result is sent to the human-computer interaction interface, and the received modified text is updated to the verification optimization text to obtain the basic colloquial text. The base colloquial text is processed by removing redundant words using a meaningless text evaluation formula to obtain preprocessed colloquial text. The preprocessed spoken text was transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanism. The standardized surgical record text is evaluated by medical record quality control experts to obtain expert evaluation results, and the standardized surgical record text is output when the expert evaluation results indicate that there are no errors.

[0006] Preferably, the formula for determining the spatial correlation of the speech slice is: ;in, The spatial correlation degree; , These are the first weight and the second weight, respectively. Indicates the percentage of intersection of feature words; This is the core feature word set of all previous pieces; This is the core feature set of all subsequent films; Indicates the time adjacency correction factor; , These are the start times for the front and back slices, respectively; This is the time decay coefficient.

[0007] Preferably, the pre-screening process for the optimal ASR model includes: Several base ASR models with different configurations were selected; the base ASR models include: a fine-tuning group and a regular group; the regular group includes: Whisper-V3, Whisper-V2, Wav2Vec2.0, and Paraformer; the fine-tuning group includes: Whisper-V2+LoRA and Paraformer+LoRA; Pre-collection of original surgical recordings dictated by the target physician during historical ophthalmic surgeries; The original surgical recording is segmented according to the natural speech pauses and the spatial correlation degree of the speech slices to obtain the original slice set; The original slice set was manually transcribed word by word and reviewed by medical record quality control experts to obtain a valid voice text set; The LoRA technique is used to fine-tune the trainable matrix in the attention layer of each model in the fine-tuning group. The error rate of spelling and the error rate of key ophthalmology terms were set as the core evaluation indicators; An independent test set was formed using several clinical surgical voice recordings of the target physicians to test each model in the conventional group and the fine-tuned group after fine-tuning, and test data was obtained. Based on the test data, the model with the best core evaluation index in the basic ASR model is set as the optimal ASR model.

[0008] Preferably, the ophthalmic surgery text determination model includes: an input layer, a lightweight BERT encoder layer, a feature extraction module, a temporal logic modeling module, a rule engine module, and an output tagging module; The input layer is connected to the lightweight BERT encoder layer; the lightweight BERT encoder layer is connected to the feature extraction module and the temporal logic modeling module; the feature extraction module is connected to the rule engine module and the output tagging module; the temporal logic modeling module is connected to the output tagging module; the rule engine module is connected to the output tagging module. The input layer is used to segment and embed the input text; the lightweight BERT encoder layer is used to extract the text semantic features of the input embedding vector and capture the association between words; the feature extraction module is used to extract the core lexical features and semantic association features of the semantic features; the temporal logic modeling module is used to capture the temporal dependencies of the text sequence; the rule engine module is used to configure and execute the judgment rules; and the output tagging module is used to locate the error position.

[0009] Preferably, the situations in which the secondary verification result does not meet the preset requirements include: the text similarity between the secondary identified text and the initial identified text exceeds the similarity threshold; or the secondary identified text is determined to be any one or a combination of two of the suspicious texts.

[0010] Preferably, the basic colloquial text is processed by removing redundant words using a meaningless text evaluation formula to obtain preprocessed colloquial text, including: Extract words from the basic colloquial text that match the pre-built meaningless vocabulary to obtain the basic eliminated and optimized text; The meaningless text evaluation formula is used to evaluate the basic text elimination and optimization to obtain a text validity score; Text segments with effective scores below the effective threshold in the basic filtered and optimized text are removed to obtain the preprocessed colloquial text.

[0011] Preferably, the formula for judging meaningless text is: ;in, To score the text effectively; The percentage of ophthalmology terms in the text segment; The first coefficient; The second coefficient; The semantic matching degree between the text fragment and the surgical procedure; This refers to text redundancy.

[0012] Preferably, the preprocessed spoken text is transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanisms, including: The pre-built colloquial text-standard medical record data is input into the local large model to learn the specific writing style, logical structure and required elements of ophthalmic surgical records; the required elements include: anesthesia method, incision preparation, surgical steps and postoperative care; The hallucination suppression mechanism is embedded in the Prompt of the fine-tuned local large model; the hallucination suppression mechanism includes: thought chain reasoning and binding instructions; The preprocessed colloquial text is input into the local large model of the hallucination suppression mechanism for standardized text generation, resulting in the standardized text of the surgical record.

[0013] Preferably, the training evaluation metrics of the local large model include: BLEU Score, ROUGE-L, and BERTScore.

[0014] Preferably, a pre-trained ophthalmic surgical text judgment model is used to verify the terminology rationality, logical coherence, and numerical validity of each initial identified text, resulting in text verification results, including: The input initial recognition text is processed sequentially using the input layer, the lightweight BERT encoder layer, and the feature extraction module to obtain the core vocabulary features; The core vocabulary features are matched with a pre-constructed ophthalmology high-frequency professional terminology database using keywords and semantic similarity calculations to obtain a vocabulary matching degree. If the vocabulary matching degree is lower than the terminology matching threshold, the output labeling module is used to label the initial identified text corresponding to the core vocabulary features as text with unreasonable terminology. The temporal logic modeling module is used to perform temporal dependency modeling and logic anomaly judgment on the output of the lightweight BERT encoder layer to obtain the logic judgment result. The output marking module is used to mark the initial recognition text corresponding to the logic judgment result being logically reversed or contradictory as logic error text. According to the pre-configured judgment rules, the rule engine module performs range judgment on the numerical values ​​extracted by the feature extraction module to obtain the numerical judgment result, and the output marking module marks the initial recognition text corresponding to the numerical judgment result that the numerical value exceeds the reasonable range as numerical error text. The text with unreasonable terminology, the text with logical errors, and the text with numerical errors are identified as the suspicious text, and the text verification result is obtained.

[0015] The present invention discloses the following technical effects: This invention provides an automatic method for generating standardized surgical record text based on speech recognition and semantic fine-tuning. By using LoRA to fine-tune two models, slice correlation analysis, multi-level verification of recognized text, text compensation recognition, and external modification text reception, it solves the problem of low accuracy in professional terminology recognition in existing methods and achieves multi-stage optimization and compensation for speech recognition. Through hallucination suppression mechanism and expert review, it solves the problem of AI hallucinations causing medical safety hazards and achieves interception of AI hallucinations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of the automatic generation process of standardized surgical record text based on speech recognition and semantic fine-tuning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of a localized ophthalmic surgical record generation system provided in an embodiment of the present invention; Figure 3 The distribution diagram of high-frequency combinations of surgical methods in the surgical recordings included in the embodiments of the present invention; Figure 4 The distribution diagram of surgical method combinations in the surgical recordings included in the embodiments of the present invention is a low-to-medium frequency distribution diagram. Figure 5 This is a comparison chart showing the performance of different ASR models provided in ophthalmic surgical scenarios according to embodiments of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide a method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning, which solves the problems of low accuracy in professional terminology recognition, insufficient ability to handle unstructured scenarios, and potential medical safety hazards caused by AI illusions in existing methods.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 This is a schematic diagram of the automatic generation process of standardized surgical record text based on speech recognition and semantic fine-tuning provided in an embodiment of the present invention, such as... Figure 1 As shown, this invention provides a method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning, comprising: Step 100: Collect the oral recording of the target physician during the operation to obtain the recording to be identified, and segment the recording to be identified according to the natural speech pauses to obtain the initial speech segments; Step 200: Calculate the spatial correlation degree of adjacent initial speech segments using the speech segment spatial correlation degree judgment formula to obtain the spatial correlation degree, and splice the initial speech segments whose spatial correlation degree exceeds a set threshold to obtain a speech segment set; Step 300: Perform speech recognition on the speech slice set using the pre-selected optimal ASR model to obtain the initial recognized text; the optimal ASR model includes: Whisper-V2+LoRA; Step 400: Use the pre-trained ophthalmic surgery text judgment model to verify the terminology rationality, logical coherence, and numerical validity of each initial identified text, and obtain the text verification results; Step 500: Mark the initial recognized text with erroneous content as suspicious text, and input the speech slice corresponding to the suspicious text and the adjacent speech slices into the optimal ASR model for secondary recognition to obtain the secondary recognized text; Step 600: Update the secondary recognition text to the initial recognition text to obtain the verification optimized text. Use the ophthalmic surgery text judgment model to verify the verification optimized text to obtain the secondary verification result. If the secondary verification result meets the preset requirements, set the verification optimized text as the basic colloquial text. Step 700: When the secondary verification result does not meet the preset requirements, send the text corresponding to the non-compliance of the secondary verification result to the human-computer interaction interface, and update the received modified text to the verification optimization text to obtain the basic colloquial text; Step 800: Use the meaningless text evaluation formula to remove redundant words from the basic colloquial text to obtain preprocessed colloquial text; Step 900: The preprocessed spoken text is transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanism. Step 1000: The standardized surgical record text is judged by the medical record quality control expert, the expert judgment result is obtained, and the standardized surgical record text when the expert judgment result is that there is no error information is output.

[0022] Specifically, the formula for determining the spatial correlation of the speech slice is as follows: ;in, The spatial correlation degree; , These are the first weight and the second weight, respectively. Indicates the percentage of intersection of feature words; This is the core feature word set of all previous pieces; This is the core feature set of all subsequent films; Indicates the time adjacency correction factor; , These are the start times for the front and back slices, respectively; This is the time decay coefficient.

[0023] Furthermore, the pre-screening process for the optimal ASR model includes: Several base ASR models with different configurations were selected; the base ASR models include: a fine-tuning group and a regular group; the regular group includes: Whisper-V3, Whisper-V2, Wav2Vec2.0, and Paraformer; the fine-tuning group includes: Whisper-V2+LoRA and Paraformer+LoRA; Pre-collection of original surgical recordings dictated by the target physician during historical ophthalmic surgeries; The original surgical recording is segmented according to the natural speech pauses and the spatial correlation degree of the speech slices to obtain the original slice set; The original slice set was manually transcribed word by word and reviewed by medical record quality control experts to obtain a valid voice text set; The LoRA technique is used to fine-tune the trainable matrix in the attention layer of each model in the fine-tuning group. The error rate of spelling and the error rate of key ophthalmology terms were set as the core evaluation indicators; An independent test set was formed using several clinical surgical voice recordings of the target physicians to test each model in the conventional group and the fine-tuned group after fine-tuning, and test data was obtained. Based on the test data, the model with the best core evaluation index in the basic ASR model is set as the optimal ASR model.

[0024] Preferably, the ophthalmic surgery text determination model includes: an input layer, a lightweight BERT encoder layer, a feature extraction module, a temporal logic modeling module, a rule engine module, and an output tagging module; The input layer is connected to the lightweight BERT encoder layer; the lightweight BERT encoder layer is connected to the feature extraction module and the temporal logic modeling module; the feature extraction module is connected to the rule engine module and the output tagging module; the temporal logic modeling module is connected to the output tagging module; the rule engine module is connected to the output tagging module. The input layer is used to segment and embed the input text; the lightweight BERT encoder layer is used to extract the text semantic features of the input embedding vector and capture the association between words; the feature extraction module is used to extract the core lexical features and semantic association features of the semantic features; the temporal logic modeling module is used to capture the temporal dependencies of the text sequence; the rule engine module is used to configure and execute the judgment rules; and the output tagging module is used to locate the error position.

[0025] Optionally, the situations in which the secondary verification result does not meet the preset requirements include: the text similarity between the secondary identified text and the initial identified text exceeds the similarity threshold; or the secondary identified text is determined to be any one or a combination of two of the suspicious texts.

[0026] Specifically, the basic colloquial text is processed by removing redundant words using a meaningless text evaluation formula to obtain preprocessed colloquial text, including: Extract words from the basic colloquial text that match the pre-built meaningless vocabulary to obtain the basic eliminated and optimized text; The meaningless text evaluation formula is used to evaluate the basic text elimination and optimization to obtain a text validity score; Text segments with effective scores below the effective threshold in the basic filtered and optimized text are removed to obtain the preprocessed colloquial text.

[0027] Furthermore, the formula for judging meaningless text is as follows: ;in, To score the text effectively; The percentage of ophthalmology terms in the text segment; The first coefficient; The second coefficient; The semantic matching degree between the text fragment and the surgical procedure; This refers to text redundancy.

[0028] Specifically, the preprocessed spoken text is transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanisms, including: The pre-built colloquial text-standard medical record data is input into the local large model to learn the specific writing style, logical structure and required elements of ophthalmic surgical records; the required elements include: anesthesia method, incision preparation, surgical steps and postoperative care; The hallucination suppression mechanism is embedded in the Prompt of the fine-tuned local large model; the hallucination suppression mechanism includes: thought chain reasoning and binding instructions; The preprocessed colloquial text is input into the local large model of the hallucination suppression mechanism for standardized text generation, resulting in the standardized text of the surgical record.

[0029] Preferably, the training evaluation metrics of the local large model include: BLEU Score, ROUGE-L, and BERTScore.

[0030] Furthermore, a pre-trained ophthalmic surgical text judgment model is used to verify the terminology rationality, logical coherence, and numerical validity of each initial identified text, resulting in text verification results, including: The input initial recognition text is processed sequentially using the input layer, the lightweight BERT encoder layer, and the feature extraction module to obtain the core vocabulary features; The core vocabulary features are matched with a pre-constructed ophthalmology high-frequency professional terminology database using keywords and semantic similarity calculations to obtain a vocabulary matching degree. If the vocabulary matching degree is lower than the terminology matching threshold, the output labeling module is used to label the initial identified text corresponding to the core vocabulary features as text with unreasonable terminology. The temporal logic modeling module is used to perform temporal dependency modeling and logic anomaly judgment on the output of the lightweight BERT encoder layer to obtain the logic judgment result. The output marking module is used to mark the initial recognition text corresponding to the logic judgment result being logically reversed or contradictory as logic error text. According to the pre-configured judgment rules, the rule engine module performs range judgment on the numerical values ​​extracted by the feature extraction module to obtain the numerical judgment result, and the output marking module marks the initial recognition text corresponding to the numerical judgment result that the numerical value exceeds the reasonable range as numerical error text. The text with unreasonable terminology, the text with logical errors, and the text with numerical errors are identified as the suspicious text, and the text verification result is obtained.

[0031] refer to Figure 2 This embodiment designs and develops a localized intelligent surgical record generation system based on an end-to-end architecture, realizing closed-loop processing of medical data in a physically isolated environment. Logically, the system is divided into three core layers: data acquisition and preprocessing layer, localized core inference layer, and human-computer interaction application layer. All data processing and model inference are completed on the hospital intranet workstation without uploading to the cloud.

[0032] Optionally, the overall system architecture is as follows: 1) Data Acquisition and Preprocessing Layer: The system acquires raw audio of doctors' intraoperative / postoperative narration using high-fidelity audio pickup equipment. An integrated VAD (Voice Activity Detection) module automatically removes silent segments, and an adaptive noise reduction algorithm filters out background noise from operating room instruments, completing audio preprocessing. A speech segment optimization module is set up to analyze the contextual relevance of audio segments using a spatial correlation judgment formula, merging segments with insufficient information to improve the quality of basic data for subsequent speech recognition. 2) Localized Core Inference Layer: A fine-tuned ASR engine and LLM generation engine based on the Transformer architecture are deployed to achieve two core transformations: from speech to text and from spoken language to standardized medical records. A full-text verification agent is set up between the ASR engine and the LLM generation engine to complete error identification, secondary correction, and removal of meaningless text from the ASR output text. Only text that passes verification can be input into the LLM generation engine, forming a three-level inference chain. 3) Human-computer interaction application layer: Provides an intuitive desktop GUI interface, integrating functions such as real-time transcription display, AI-generated content differentiation highlighting, and one-click export archiving. It also has a built-in hazard interception mechanism that automatically issues warnings when the model confidence is low, ensuring that doctors have a very low learning cost. It sets up a human-computer verification entry point for secondary correction of speech recognition, receiving highly similar secondary recognition text transmitted by the full-text verification agent, allowing doctors to quickly verify and modify it.

[0033] Optionally, given the strict restrictions on data export within hospital intranet environments and the fact that most hospitals lack large-scale computing resources, this embodiment performs model fine-tuning and local inference on consumer-grade hardware, balancing computational power requirements with clinical practicality: Computing platform: A workstation equipped with a high-performance NVIDIA RTX 5070 Ti / 5090 graphics card is selected as the local inference server, utilizing its Tensor Cores to accelerate matrix operations; Inference optimization technology: 4-bit / 8-bit quantization technology is used to compress model weights, reducing memory usage by more than 50% with almost no loss of accuracy; at the same time, vLLM or FlashAttention is used to accelerate inference throughput, ensuring that the generation speed meets the real-time requirements of clinical applications; The full-text verification agent and speech slicing optimization module are both deployed on this local computing platform, employing lightweight algorithm design without adding extra computational burden, ensuring the real-time performance of the overall system.

[0034] Specifically, data collection and preprocessing involved retrospectively collecting original surgical recordings dictated by senior surgeons during ophthalmological surgeries at the target hospital, focusing on vitreoretinal surgery and covering complex procedures such as vitrectomy (simple / combined cataract surgery), external retinal surgery, and lens suspension. The recordings were captured in .wav format using a voice memo tool. Long recordings were segmented into short segments of less than 30 seconds based on natural speech pauses and converted to .m4a format with a 16K sampling rate. A formula for judging the spatial relevance of speech segments was introduced, and a creative expression was constructed by combining the semantic relevance and temporal adjacency of the segment context to address the issues of insufficient information and missing context caused by natural speech pause segmentation. The specific expression is as follows: .

[0035] in, The final spatial correlation degree is not less than the threshold. Simultaneous merging of slices, The original segmentation is retained at the time; The proportion of the intersection of core feature words, This is the core feature set of all previous films. The core feature word set for subsequent slices consists of high-frequency ophthalmic surgical terms and surgical operation keywords, which are extracted from the initial speech-to-text draft using the TF-IDF algorithm. This is the time adjacency correction factor. , These represent the start times of the front and back slices, respectively. This is the time decay coefficient; , These are the weighting coefficients. It has a higher weight and prioritizes the core role of semantic relevance, while assisting in correcting the impact of temporal adjacency.

[0036] refer to Figure 3 and Figure 4 A total of 523 surgical recordings were included, resulting in 1492 audio segments (approximately 14 hours in total duration). All segments underwent both manual transcription and review by medical record quality control experts. Transcription personnel strictly recorded the exact vocabulary (including spoken language) without any manual corrections. After spatial correlation analysis and merging, 1381 valid audio segments were ultimately obtained, effectively addressing the issue of insufficient information from a single segment. Future research will continue to expand the data collection scope, incorporating more doctors' accents and diverse surgical procedures to construct a high-quality ophthalmic audio dataset consisting of a triplet of audio, raw text, and standard medical records.

[0037] Preferably, the selection of ASR models and LoRA fine-tuning are performed. Model comparison targets: Five different configurations of ASR models were selected for performance comparison, including Whisper-V3, Whisper-V2, Whisper-V2+LoRA, Paraformer (Alibaba), Paraformer+LoRA, and Wav2Vec2.0 (Meta). For the fine-tuning group (Whisper+LoRA and Paraformer+LoRA), low-rank adaptation (LoRA) technology was used. A trainable matrix was injected into the attention layer of the pre-trained model, freezing most parameters and training only the inserted low-rank matrix. This efficiently adapts to ophthalmological terminology and doctors' individual speaking habits while avoiding catastrophic forgetting. Transcription accuracy was measured using word error rate (CER) and key ophthalmological terminology error rate (KTER). A separate test set of 100 clinical surgical voice recordings (completely isolated from the training set) was selected, and the model with the lowest CER was chosen as the front-end input for the subsequent document generation module.

[0038] CER: Based on edit distance, the formula is: .

[0039] Where S = Replacement, D = Deletion, I = Insertion, and N = Total number of characters in the standard reference text, the lower the value, the more accurate the dictation. This embodiment aims to achieve... ; KTER: Construct an independent test set containing 100 high-frequency ophthalmology terms, and count the proportion of key terms that are misidentified. The lower the value, the higher the reliability of the model in terms of professional context.

[0040] refer to Figure 5 Model performance conclusions: Whisper-V2+LoRA performed best. The original model had a high CER, but after fine-tuning with LoRA, it showed great flexibility, minimizing various error rates. Paraformer, as the optimized Chinese model, had fast inference speed, achieving a KTER of 4.89% after fine-tuning, demonstrating excellent overall performance. Wav2Vec2.0, without targeted fine-tuning, had a KTER as high as 81.98%, but could not handle ophthalmological terminology. The optimal ASR model selected in this embodiment provides basic model support for the secondary recognition stage of the full-text verification agent. The parameter configuration of this optimal model is used during secondary recognition to ensure recognition consistency.

[0041] Furthermore, to address issues such as errors and meaningless information in the output text of speech recognition models, a full-text verification agent is constructed between the ASR model and the LLM model. This agent is built on a lightweight BERT model, fine-tuned for ophthalmic surgery scenarios, retaining only the core functions of text semantic recognition, error detection, and keyword matching. It is deployed on a local computing platform and does not rely on cloud computing power. The core workflow of the full-text verification agent is as follows: S1: Receives the output text from the optimal ASR model, and uses a pre-trained ophthalmic surgery text judgment model based on a lightweight BERT architecture, using BERT-base-uncased as the pre-training base, fine-tuned for ophthalmic surgery scenarios. The core network architecture and module composition are as follows: Input layer: Responsible for receiving the ASR output text, completing text segmentation, word embedding, and position encoding, converting the text into a vector form that the model can process; Lightweight BERT encoder layer: Retains 6 Transformer encoder layers (removing the original 8-layer encoder from BERT-base to reduce computational power consumption), each layer containing a multi-head attention machine. The system employs a 4-head attention architecture and a Feed-Forward network to extract semantic features from the text and capture the relationships between words. A feature extraction module, connected to the BERT encoder layer, is responsible for dimensionality reduction and filtering of the encoded semantic features, extracting core lexical features and semantic association features from the text. A temporal logic modeling module embeds a BiLSTM layer (2 layers) to capture the temporal dependencies of the text sequence. A rule engine module has a built-in configurable rule base interface for presetting judgment rules such as reasonable ranges for clinical values ​​and terminology matching thresholds. An output labeling module receives the judgment results from each module and locates errors using coordinate labeling. The specific correspondence and judgment methods include: 1) Terminology rationality: Relying on the model's input layer, lightweight BERT encoder layer, and feature extraction module, the input layer segments the text and converts it into word vectors. The encoder layer extracts semantic features of words, and the feature extraction module selects core word features. Subsequently, keyword matching and semantic similarity calculation (completed by the cosine similarity calculation unit) are performed with the model's built-in pre-built ophthalmology high-frequency professional terminology database. If the model determines that the word matching degree is lower than the preset threshold and there are no synonymous terms, the output labeling module marks the terminology as unreasonable; 2) Logical coherence: Relying on the model's temporal logic modeling module (BiLSTM layer) and lightweight BERT encoder layer, the encoder layer extracts semantic association features of words related to surgical steps, and BiLS... The TM layer captures the temporal dependencies of words in each surgical step. Combined with the ophthalmic surgical operation logic library implanted during model training, the BiLSTM layer analyzes whether the order of surgical steps in the text conforms to clinical norms, such as making an incision before vitrectomy, or exploring before deciding on the surgical plan. If logical inversion or contradiction is identified, the signal is passed to the output marking module and marked as a logical error. 3) Numerical validity: Relying on the feature extraction module and the rule engine module, the feature extraction module extracts all numerical values ​​and corresponding unit features from the text and passes them to the rule engine module. The rule engine module calls the preset reasonable range of key ophthalmic clinical values ​​and performs threshold verification on the values. If it is determined that the value exceeds the reasonable range or the numerical format is incorrect, the output marking module marks it as a numerical error.After recognition is completed, the model's output labeling module marks the location and range of the erroneous content using coordinate labeling, extracts the closest speech slices above and below the erroneous content, and combines them into a new overall slice.

[0042] S2: Secondary Speech Recognition: The combined new slice is input into the optimal ASR model for re-recognition. The output of the secondary recognition replaces the marked erroneous content in the original text, completing the first error correction. After correction, the entire text is checked again. If the check result meets the preset pass criteria, the text is deemed to have passed the check and can be input into the subsequent LLM model.

[0043] S3: If the text similarity between the secondary recognition result and the first recognition result is ≥80% (calculated by the cosine similarity algorithm), or if the text updated by the secondary speech recognition still has text that does not meet the requirements after verification by the ophthalmic surgery text judgment model, but the similarity between this text and the corresponding text of the initial recognition is less than 80%, then it is determined that this part of the content cannot be automatically corrected by the model. The marked part of the text is pushed to the human-computer interaction interface, marked as requiring manual verification. The doctor will quickly check and modify it. The text confirmed by the doctor will be automatically sent back to the full-text verification agent to complete the correction.

[0044] S4: In the above verification process, basic meaningless word removal is first performed: words in the identified text that completely match a pre-built meaningless vocabulary database are removed. This vocabulary database includes common interjections, redundant particles, and irrelevant oral expressions found in clinical speech. After completing the basic filtering, the ASR output text is further filtered using a pre-set meaningless text evaluation formula to remove colloquial meaningless words, interjections, and repetitive expressions. The specific expression is as follows: .

[0045] in, To score the text effectively, Effective threshold Text deemed meaningless was removed. Then retain; The percentage of ophthalmology terms in the text segment; The semantic matching degree between the text fragment and the surgical procedure; To reduce text redundancy, redundant expressions are removed. , All are weighting coefficients.

[0046] After the above steps, the full-text verification agent outputs reliable spoken text that is error-free, non-redundant, and with complete context, which serves as input data for the subsequent LLM model, thus improving the text processing quality of the LLM model from the source.

[0047] Specifically, the Qwen3-8B open-source large language model was selected as the core of semantic understanding. This model demonstrates excellent instruction following and text reconstruction capabilities in specific domain tasks. The input data for this model is standardized spoken text processed by a full-text verification agent, which significantly improves the quality of the input data compared to directly receiving the output of an ASR model. Qwen3-8B was deployed on a local server, and the deployed model was fine-tuned using LoRA instructions with constructed spoken text-standard medical records. The model was trained to learn the specific writing style, logical structure, and required elements (anesthesia method, incision preparation, surgical steps, postoperative care, etc.) of ophthalmic surgical records. Difficult samples were introduced for training on ocular trauma and complex vitreoretinal surgeries to improve the model's ability to handle non-standardized and non-linear surgical procedures.

[0048] Preferably, the hallucination suppression mechanism involves introducing CoT reasoning and binding instructions into the Prompt, forcing the model to generate content solely based on the input text, strictly prohibiting the fabrication of unmentioned surgical procedures, and ensuring the absolute accuracy of medical facts. Because the input text undergoes full-text verification by an intelligent agent to remove meaningless content and correct errors, the probability of hallucinations caused by input data bias is further reduced. The model receives the colloquial text output by the full-text verification agent, and while maintaining medical semantic accuracy, corrects grammatical errors, removes colloquial redundancy, and fills in logical gaps, reconstructing it into a structured and standardized medical document.

[0049] Referring to Table 1, three complementary NLP metrics—BLEU-4, ROUGE-L, and BERTScore—are introduced to quantify the linguistic similarity between AI-generated text and expert-written standard text. Each metric focuses on different dimensions, forming a comprehensive evaluation. Because the text input to the LLM model is optimized by a full-text validation agent, the evaluation benchmarks of these three metrics are further improved in this embodiment, achieving the desired goal. , , .

[0050] Table 1

[0051] Preferably, the generated documents are reviewed by medical record quality control experts. A binary scoring method is used to determine whether key information such as eye type and intraocular lens status is correct or incorrect. Simultaneously, errors in the AI-generated content are subdivided into three categories for precise analysis of model defects: 1) Ambiguity / Misunderstanding: The model fails to correctly interpret unclear expressions in the spoken description, such as misinterpreting minimally invasive three-channel surgery as other specific procedures; after processing by the full-text verification agent, the incidence of this type of error is reduced by more than 90%. 2) Factual Errors: The model generates incorrect data or facts, such as recording 10mmHg as 20mmHg or incorrectly outputting energy points; after numerical validity verification by the full-text verification agent, the incidence of this type of error is reduced to 0. 3) Hallucinations: The model generates surgical steps, medications, or signs not mentioned in the recording, such as automatically completing the missing posterior capsule resection step. The dual processing of speech segmentation optimization and the full-text verification agent reduces the triggering factors for model hallucinations at the input data level.

[0052] Optionally, referencing the AHRQ (American Health Organization for Research and Quality) guidelines, a hazard assessment scale is introduced. Experts rate each identified error in a two-dimensional manner to assess the actual impact of potential AI errors on patient safety, while simultaneously verifying whether AI's rationalization illusion increases hidden medical risks: 1) Severity of hazard: If the error is not detected and enters the medical record, the possible consequences are categorized as no harm, mild, moderate, severe, life-threatening, or blinding; 2) Probability of harm: The probability that the error will cause the above harms is categorized as low, moderate, or high. Because the full-text verification agent intercepts most of the erroneous content in advance, the probability of severe / moderate harm in the text processed by the system in this embodiment is reduced to 0.

[0053] The beneficial effects of this invention are as follows: This invention fine-tunes two models using LoRA, improving their ability to recognize terminology and process unstructured text in ophthalmic scenarios. Through slice correlation analysis and multi-level verification of recognized text, it enhances the reliability and accuracy of text recognition in professional scenarios for the LoRA-tuned ASR model. By using text compensation recognition and externally modified text reception, it improves the accuracy of text recognition and avoids inputting suspicious content into subsequent steps. Through hallucination suppression mechanisms and expert review, it intercepts AI hallucinations, ensuring the standardization and accuracy of surgical records.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0055] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning, characterized in that, include: The oral recording of the target physician during surgery is collected to obtain the recording to be identified. The recording to be identified is segmented according to the natural speech pauses to obtain the initial speech segments. The spatial correlation degree of adjacent initial speech segments is calculated using the speech segment spatial correlation degree judgment formula to obtain the spatial correlation degree, and the initial speech segments with spatial correlation degree exceeding a set threshold are spliced ​​together to obtain a speech segment set. The pre-selected optimal ASR model is used to perform speech recognition on the speech slice set to obtain the initial recognition text; The optimal ASR model includes: Whisper-V2+LoRA; The pre-trained ophthalmic surgery text judgment model is used to verify the terminology rationality, logical coherence, and numerical validity of each initial identified text, and the text verification results are obtained. The initial recognized text whose text verification result is incorrect is marked as suspicious text, and the speech slice corresponding to the suspicious text and the adjacent speech slices are combined and input into the optimal ASR model for secondary recognition to obtain the secondary recognized text; The secondary recognition text is updated to the initial recognition text to obtain the verification optimized text. The verification optimized text is then verified using the ophthalmic surgery text judgment model to obtain the secondary verification result. If the secondary verification result meets the preset requirements, the verification optimized text is set as the basic colloquial text. When the secondary verification result does not meet the preset requirements, the text corresponding to the non-compliance in the secondary verification result is sent to the human-computer interaction interface, and the received modified text is updated to the verification optimization text to obtain the basic colloquial text. The base colloquial text is processed by removing redundant words using a meaningless text evaluation formula to obtain preprocessed colloquial text. The preprocessed spoken text was transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanism. The standardized surgical record text is evaluated by medical record quality control experts to obtain expert evaluation results, and the standardized surgical record text is output when the expert evaluation results indicate that there are no errors.

2. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The formula for determining the spatial correlation of the speech slice is: ;in, The spatial correlation degree; , These are the first weight and the second weight, respectively. Indicates the percentage of intersection of feature words; This is the core feature word set of all previous pieces; This is the core feature set of all subsequent films; Indicates the time adjacency correction factor; , These are the start times for the front and back slices, respectively; This is the time decay coefficient.

3. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The pre-screening process for the optimal ASR model includes: Several base ASR models with different configurations were selected; the base ASR models include: a fine-tuning group and a regular group; the regular group includes: Whisper-V3, Whisper-V2, Wav2Vec2.0, and Paraformer; the fine-tuning group includes: Whisper-V2+LoRA and Paraformer+LoRA; Pre-collection of original surgical recordings dictated by the target physician during historical ophthalmic surgeries; The original surgical recording is segmented according to the natural speech pauses and the spatial correlation degree of the speech slices to obtain the original slice set; The original slice set was manually transcribed word by word and reviewed by medical record quality control experts to obtain a valid voice text set; The LoRA technique is used to fine-tune the trainable matrix in the attention layer of each model in the fine-tuning group. The error rate of spelling and the error rate of key ophthalmology terms were set as the core evaluation indicators; An independent test set was formed using several clinical surgical voice recordings of the target physicians to test each model in the conventional group and the fine-tuned group after fine-tuning, and test data was obtained. Based on the test data, the model with the best core evaluation index in the basic ASR model is set as the optimal ASR model.

4. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The ophthalmic surgery text determination model includes: an input layer, a lightweight BERT encoder layer, a feature extraction module, a temporal logic modeling module, a rule engine module, and an output tagging module; The input layer is connected to the lightweight BERT encoder layer; the lightweight BERT encoder layer is connected to the feature extraction module and the temporal logic modeling module; the feature extraction module is connected to the rule engine module and the output tagging module; the temporal logic modeling module is connected to the output tagging module; the rule engine module is connected to the output tagging module. The input layer is used to segment and embed the input text; the lightweight BERT encoder layer is used to extract the text semantic features of the input embedding vector and capture the association between words; the feature extraction module is used to extract the core lexical features and semantic association features of the semantic features; the temporal logic modeling module is used to capture the temporal dependencies of the text sequence; the rule engine module is used to configure and execute the judgment rules; and the output tagging module is used to locate the error position.

5. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The situations in which the secondary verification result does not meet the preset requirements include: the text similarity between the secondary identified text and the initial identified text exceeds the similarity threshold; and the secondary identified text is determined to be any one or a combination of two of the suspicious texts.

6. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The basic colloquial text is processed by removing redundant words using a meaningless text evaluation formula to obtain preprocessed colloquial text, including: Extract words from the basic colloquial text that match the pre-built meaningless vocabulary to obtain the basic eliminated and optimized text; The meaningless text evaluation formula is used to evaluate the basic text elimination and optimization to obtain a text validity score; Text segments with effective scores below the effective threshold in the basic filtered and optimized text are removed to obtain the preprocessed colloquial text.

7. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The formula for judging meaningless text is: ;in, To score the text effectively; The percentage of ophthalmology terms in the text segment; The first coefficient; The second coefficient; The semantic matching degree between the text fragment and the surgical procedure; This refers to text redundancy.

8. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The preprocessed spoken text is transformed into structured, standardized surgical record text using a local large model with LoRA-tuned embedded hallucination suppression mechanisms, including: The pre-built colloquial text-standard medical record data is input into the local large model to learn the specific writing style, logical structure and required elements of ophthalmic surgical records; the required elements include: anesthesia method, incision preparation, surgical steps and postoperative care; The hallucination suppression mechanism is embedded in the Prompt of the fine-tuned local large model; the hallucination suppression mechanism includes: thought chain reasoning and binding instructions; The preprocessed colloquial text is input into the local large model of the hallucination suppression mechanism for standardized text generation, resulting in the standardized text of the surgical record.

9. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 1, characterized in that, The training evaluation metrics for the local large model include: BLEU Score, ROUGE-L, and BERTScore.

10. The method for automatically generating standardized surgical record text based on speech recognition and semantic fine-tuning according to claim 4, characterized in that, A pre-trained ophthalmic surgery text judgment model is used to verify the terminology rationality, logical coherence, and numerical validity of each initially identified text, resulting in text verification results, including: The input initial recognition text is processed sequentially using the input layer, the lightweight BERT encoder layer, and the feature extraction module to obtain the core vocabulary features; The core vocabulary features are matched with a pre-constructed ophthalmology high-frequency professional terminology database by keyword matching and semantic similarity calculation to obtain the vocabulary matching degree. If the vocabulary matching degree is lower than the terminology matching threshold, the output marking module is used to mark the initial identified text corresponding to the core vocabulary features as text with unreasonable terminology. The temporal logic modeling module is used to perform temporal dependency modeling and logic anomaly judgment on the output of the lightweight BERT encoder layer to obtain the logic judgment result. The output marking module is used to mark the initial recognition text corresponding to the logic judgment result being logically reversed or contradictory as logic error text. According to the pre-configured judgment rules, the rule engine module performs range judgment on the values ​​extracted by the feature extraction module to obtain the value judgment result, and the output marking module marks the initial recognition text corresponding to the value exceeding the reasonable range as the value error text. The text with unreasonable terminology, the text with logical errors, and the text with numerical errors are identified as the suspicious text, and the text verification result is obtained.

Images