A multi-level ASR cooperation-based reproductive cycle recording device and method

Abstract

The application discloses a kind of based on multistage ASR cooperation's reproductive cycle of promoting and recording device and method, it is related to medical information and artificial intelligence cross technical field, including interface learning module, adaptive learning module, context management module, user interaction module, core configuration module, voice wake-up module, voice recognition module, instruction set model, free text model, text proofreading module and interface operation module;The present application overcomes the existing medical record system in the monitoring of assisted reproductive ovulation cycle The deficiencies of low voice input efficiency, high cost of third-party system integration, lack of context memory and adaptive learning ability, with the ability of collaborative processing multiple types of voice input, AI dynamically adapts heterogeneous system interface, context memory and adaptive learning.

Description

Technical Field

[0001] This invention relates to the field of medical informatics and artificial intelligence, specifically to a reproductive ovulation induction cycle recording device and method based on multi-level ASR collaboration. Background Technology

[0002] In the clinical practice of assisted reproductive technology, monitoring patients' ovulation induction cycles (such as follicle development, endometrial thickness, hormone levels, etc.) is a core aspect of diagnosis and treatment, requiring doctors to frequently and extensively input and record data. Currently, the mainstream recording methods are still mainly manual keyboard input or voice input systems relying on dedicated interfaces, which have the following technical shortcomings: First, existing speech recognition systems mostly use a single, general-purpose model, lacking a multi-layered language model collaborative design specifically for ovulation induction cycle monitoring scenarios. The system cannot uniformly handle two types of speech input: control commands such as "save record" and "next page," and free text descriptions such as "a 18mm follicle is visible in the right ovary." This often requires calling different modules separately, increasing processing latency and leading to mis-triggered commands or loss of text semantics due to the lack of interaction between models, severely impacting the fluency and accuracy of input.

[0003] Secondly, the integration of existing voice input systems with third-party electronic record systems mostly relies on dedicated API interfaces, lacking the ability to dynamically recognize and directly manipulate interface elements of third-party medical systems, such as simulating manual clicks and filling in information. For older systems without open interfaces or with limited interface functionality, integration is impossible, resulting in poor universality; even with interfaces, custom development is required, leading to long development cycles, high costs, and difficulty in rapid replication across different hospitals.

[0004] Furthermore, doctors' voice input during diagnosis and treatment often involves multi-turn dialogues, including references (such as "the one on the left") and omissions (such as "what about size?"). Existing systems lack dialogue state tracking and contextual memory, making it impossible to understand the semantic meaning of subsequent speech, leading to duplicate information input or incorrect retrieval. At the same time, the system lacks adaptive learning capabilities, failing to learn personalized input habits from doctors' historical operations, requiring each input to start from scratch, thus failing to truly reduce the workload of doctors. Summary of the Invention

[0005] To address the aforementioned shortcomings of the prior art, this invention provides a reproductive ovulation induction cycle recording device and method based on multi-level ASR coordination, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a reproductive ovulation induction cycle recording device based on multi-level ASR collaboration, comprising an interface learning module, an adaptive learning module, a context management module, a user interaction module, a core configuration module, a voice wake-up module, a voice recognition module, an instruction set model, a free text model, a text proofreading module, and an interface operation module. The interface learning module is used to connect with the target third-party medical system. It extracts control features by parsing the interface files and builds an operation mapping library. The adaptive learning module is connected to the speech recognition module and the interface learning module respectively. It dynamically judges the matching of speech input type, generates incremental training data by manually correcting behavior, and processes the input type. The context management module is connected to the adaptive learning module and the free text model respectively, and is controlled by the core configuration module. It is used to selectively enable context memory according to the preset memory function on / off state. The user interaction module provides a visual interaction interface between doctors and the system, enabling operation control, status display, result preview, and log recording. The core configuration module is used to centrally manage all configuration behavior switches in the system. It is responsible for receiving preset and dynamic adjustment instructions from doctors, maintaining the real-time status of switch parameters, and synchronizing the parameters to various business modules through a unified interface. The voice wake-up module is used to monitor the microphone audio stream in real time, recognize the preset wake-up word, and trigger the system to enter the formal voice recognition process. The speech recognition module is used to receive the speech stream from the voice wake-up module and complete audio preprocessing and speech feature extraction; The instruction set model is used to parse operation instructions and output standardized instruction IDs and confidence scores when the adaptive learning module determines that the input is an instruction class. The free text model is used to identify and output standardized text and extract structured medical entities when the adaptive learning module determines that the input is free text. The text proofreading module is used to perform secondary processing on the output of the instruction set model or free text model throughout the entire process. The interface operation module is connected to the interface learning module and the text proofreading module respectively. Based on the operation intention and the operation mapping library, it adopts a three-level positioning strategy to achieve accurate positioning of third-party medical system controls, and simulates manual operation through the system-level interface to complete interface-free data entry and system control.

[0007] Preferably, the interface learning module includes: The export unit is used to export the HTML file of the target third-party medical system or generate a high-resolution interface snapshot. The generation unit is used to export the interface file and save it in a standard format that the interface learning module can parse. The parsing unit is used to read and parse standard format interface files to complete the identification of interface structure and elements; The feature extraction unit is used to extract the control type, screen coordinates, unique identifier, associated text label, and executable operation; The mapping library building unit is used to organize the extracted control features into a structured operation mapping library in JSON format, establishing a one-to-one correspondence between interface elements and system operation commands.

[0008] Preferably, the adaptive learning module includes input type matching, human-computer interaction closed-loop optimization, and personalized model fine-tuning; Input type matching: Receives speech features from the speech recognition module, combines them with the operation mapping library of the interface learning module, and dynamically determines whether the speech input type is command type or free text type; Human-computer interaction closed-loop optimization: Record doctors' manual correction behavior to the system output, generate incremental training data, and provide samples for personalized model fine-tuning; Personalized model fine-tuning: Utilize the correction data accumulated through closed-loop optimization of human-computer interaction to periodically perform incremental updates on the feature vector space of the instruction set model or lightweight fine-tuning on the free text model.

[0009] Preferably, the input type is determined by calculating the semantic distance between the speech content and the instruction class features, with the type classification threshold set to 0.8. The specific determination is as follows: If the probability of matching the speech content with the command-type feature is ≥0.8, it is directly determined to be a command-type feature; If the matching probability with the free text category feature is ≥0.8, it is directly determined to be a free text category; If both are below 0.8, a comprehensive judgment process is initiated, and a secondary decision is made by combining the context and historical operating patterns.

[0010] Preferably, the context management module includes context memory management and reference resolution and information completion. When context memory is enabled: Context memory management: The system automatically maintains a dialogue state cache, recording the most recently identified ovulation induction monitoring medical entities and dialogue states; Substitution resolution and information completion: The recognition results output by the free text model are processed, semantic completion is performed on speech with semantic reference or semantic reference, and the completed semantic information is passed to the interface operation module.

[0011] Preferably, the referential resolution and information completion specifically include: If the current input contains a directional word, then find the most recent record containing that directional word in the context cache and complete it. If the current input contains a vague pronoun, return the most recent record from the context cache; If the current input is an elliptical sentence, then the missing numeric field is detected, and the most recent record matching the current scenario is searched from the context cache to complete it; If the current input is only a numerical value, and the metrics entered in the previous round match the current scenario, they will be automatically combined into a complete structured entity.

[0012] Preferably, the parameter switch includes: Memory function switch: Boolean, defaults to true, controls whether to enable dialogue state tracking and context reuse; Model switching switch: enumeration type, optional values ​​are auto / command / text, default is auto, which affects the decision logic of input type judgment; Proofreading function switch: Boolean, defaults to true, controls whether the text proofreading model is enabled.

[0013] Preferably, the operational intent specifically includes: Data population: Mapping structured medical data to the corresponding record fields in the target system; System operations: including executing system commands such as save, page turning, and page switching; Information Supplement: If the recognition result is incomplete, the user interaction module will prompt the doctor to supplement relevant diagnosis and treatment information.

[0014] Preferably, the three-level positioning strategy is prioritized as follows: precise ID matching, fuzzy text label matching, and coordinate region positioning. It calls the Windows SendInput / UIAutomation operating system automation interface to simulate mouse movement, clicks, keyboard input, and focus switching.

[0015] A method for recording ovulation induction cycles based on multi-level ASR collaboration is applied to a ovulation induction cycle recording device based on multi-level ASR collaboration, including an offline learning phase and an online usage phase: The offline learning phase includes the following steps: A1: Export the interface files of the target third-party medical system through the interface learning module; A2: Parse the interface file and identify interactive controls in the interface; A3: Extract the core features of each control; A4: Organize the extracted control features into a structured operation mapping library; A5: Perform simulated positioning and operation tests on all controls in the operation mapping library; The online usage phase includes the following steps: B1: Doctors log in to the system through the user interaction module, and the core configuration module loads and synchronizes personalized switch parameters to each execution unit; B2: The system uses low-power listening for wake words. After a successful match, it enters recognition mode and collects the voice stream in real time. B3: The speech recognition module extracts features from the speech stream and passes them to the adaptive learning module for input type discrimination; B4: Based on the input type discrimination result, the system calls the corresponding model to complete speech recognition and outputs standardized results; B5: When the memory function is enabled in the core configuration module, the context management module maintains the dialogue state and completes the referential resolution and information completion. B6: The text proofreading module performs terminology correction, format standardization, and numerical logic verification on the recognition results, and outputs the final standardized and structured data; B7: The system integrates the current recognition results, context information, and semantics of the operation mapping library to parse out three types of operation intentions; B8: The interface operation module retrieves controls from the operation mapping library through a three-level positioning strategy, calls the automation interface to execute operations, and supports retries; B9: The user interaction module displays the operation results in real time, allowing doctors to check and correct them. The system records the correction behavior for subsequent optimization. B10: The adaptive learning module records data pairs before and after correction, and achieves personalized optimization and closed-loop feedback through incremental updates.

[0016] This invention provides a reproductive ovulation induction cycle recording device and method based on multi-level ASR coordination, which has the following beneficial effects: 1. This invention utilizes an adaptive learning module and a dual-model (instruction set model and free text model) collaborative invocation mechanism to achieve accurate separation of instruction-based and free text-based speech based on interface semantic anchors and feature similarity calculation (threshold 0.8). Compared to a single general model, this effectively avoids the problems of accidental instruction triggering and loss of text semantics. Furthermore, the three-level processing (terminology correction, format standardization, and numerical logic verification) of the text proofreading module further ensures the quality of the input data. 2. This invention uniquely employs an interface learning module. By exporting the target system's HTML file or interface snapshot, and combining it with DOM parsing or YOLOv5 object detection, it automatically extracts control features and constructs a JSON-formatted operation mapping library. The interface operation module, based on a three-level positioning strategy (exact ID matching → fuzzy text tag matching → coordinate region positioning), directly calls the operating system's automated interface to simulate manual operation, achieving "interface-free" integration. This invention requires no modification to the target medical system, shortening the traditional system integration cycle of weeks or even months to hours, significantly reducing deployment and maintenance costs, and possessing strong cross-platform versatility. 3. When the memory function is enabled, the context management module of this invention automatically maintains a dialogue state cache with a capacity of 10 rounds. Based on rule engines such as locative word matching, fuzzy referential matching, elliptical sentence completion, and pure numerical inheritance, it achieves intelligent referential resolution and information completion. During continuous data entry, doctors only need to verbally state "What about the left side? 16 mm" to automatically inherit the semantics of "follicle size" from the previous round, without needing to repeat the complete description.

[0017] 4. The adaptive learning module of this invention records the doctor's manual corrections to the system output (type judgment, recognized content) in real time, generates incremental training data, and incrementally updates the instruction feature vector space or performs lightweight fine-tuning on the free text model during idle periods or after accumulating a threshold. This closed-loop feedback mechanism enables the system to gradually adapt to the doctor's personalized language habits and accent characteristics, continuously improving recognition accuracy and user satisfaction. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating the overall system module architecture and data flow of the present invention. Figure 2 This is a diagram showing the internal structure of the adaptive learning module of the present invention; Figure 3 This is a decision logic diagram for the three-level positioning strategy of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The present invention will now be described in detail through specific embodiments, as follows: According to the instruction manual Figures 1-3As can be seen, when the present invention is in operation, a reproductive ovulation induction cycle recording device based on multi-level ASR collaboration includes an interface learning module, an adaptive learning module, a context management module, a user interaction module, a core configuration module, a voice wake-up module, a voice recognition module, an instruction set model, a free text model, a text proofreading module, and an interface operation module.

[0021] The interface learning module (static configuration layer) is responsible for connecting to the target third-party medical system. It extracts control features by parsing HTML / images and builds an initial "operation mapping library".

[0022] The interface learning module includes an export unit, a generation unit, a parsing unit, a feature extraction unit, and a mapping library construction unit. The specific workflow is as follows: First, in the export unit, the interface files of the target third-party medical system are exported using automated tools (such as Puppeteer for web applications and UI Automation API for desktop applications). In the generation unit, for web applications, the complete HTML file of the current page is exported; for desktop applications, a high-resolution (e.g., 1920×1080) PNG snapshot of the interface is generated.

[0023] Then, in the parsing unit, a differentiated strategy is used to parse the interface file: the web application traverses the document object model through a DOM parser (such as jsdom) to extract all interactive elements (such as... <input> , <button>The UI Automation API is used to obtain the control's ID, name, type, associated text label, and screen coordinates. For desktop applications, the UI Automation API is called first to obtain the control's properties. If the target application does not support it, the pre-trained YOLOv5 object detection model is used as an alternative to identify the control's type and position.

[0024] Next, in the feature extraction unit, the core features of each control are extracted from the parsing results, including control type, screen coordinates, unique identifier, associated text label, and executable trigger logic (such as "click" and "input").

[0025] Finally, in the mapping library construction unit, these control features are organized according to the target system's window-page-control hierarchy, generating a structured operation mapping library in JSON format, establishing a one-to-one correspondence between interface elements and system operation commands. An example mapping library is as follows: json { "window": "Reproductive Center Record System", "pages": [{ "page_name": "Ultrasound recording page", "controls": [ {"id": "txtFollicle", "type": "edit", "text": "follicle size", "rect": {"x":150,"y":250, "width":100, "height":30}, "action": "set"}, {"id": "btnSave", "type": "button", "text": "Save", "rect": {"x":200,"y":300, "width":80, "height":30}, "action": "click"} ] }] }

[0026] It is important to understand that third-party medical systems include, but are not limited to, assisted reproductive electronic medical record systems, outpatient electronic medical record systems, LIS, PACS and other clinical information systems already deployed in hospitals. These are essential and readily available software in every hospital, providing external data sources and operating interfaces for this invention. Ultimately, the data must be stored in the aforementioned systems to enable the operation of clinical business processes.

[0027] The adaptive learning module (dynamic adaptation layer) is one of the core components of this device, responsible for online input type judgment, human-computer interaction closed-loop optimization, and personalized model fine-tuning. The adaptive learning module connects to both the speech recognition module and the interface learning module, dynamically judging the matching of speech input types, generating incremental training data through manual behavior correction, and processing the input types.

[0028] The adaptive learning module includes input type matching, human-computer interaction closed-loop optimization, and personalized model fine-tuning. The specific workflow is as follows: First, after receiving the speech features, the adaptive learning module extracts the list of control text labels (such as "Save", "Cancel", "Next Page", "Follicle Size") of the current page from the operation mapping library built by the interface learning module as interface semantic anchors.

[0029] Then, input type matching is performed. The system calculates the similarity (cosine similarity) between the speech features and the instruction feature template library to obtain the instruction class matching probability simmax; simultaneously, the speech features are input into a pre-trained SVM free text classifier to obtain the free text matching probability Ptext. The input type is determined by simmax and Ptext.

[0030] Next, the human-computer interaction closed-loop optimization is performed. The adaptive learning module also records every manual correction made by the doctor to the system output (such as type judgment and recognized content), generates incremental training data containing the original input and the corrected labels, and stores it in the local database.

[0031] Finally, personalized model fine-tuning is performed. The system has a built-in timer that, during off-peak hours (such as at night) or after the accumulated samples reach a threshold, uses this data to incrementally update the instruction feature vector space or fine-tune the free text model, allowing the system to gradually adapt to the doctor's personalized language habits and achieve personalized adaptation.

[0032] The context management module (memory and completion) is connected to the adaptive learning module and the free text model respectively, and is controlled by the "memory function switch" of the core configuration module. It is used to selectively enable context memory according to the preset memory function switch state, and perform full-process secondary processing on the output results of the instruction set model or the free text model.

[0033] When context memory is enabled (i.e., when the switch is true), the system automatically maintains a dialogue state cache (capacity 10 rounds). When the free text model outputs structured medical entities, this module stores them in the cache and updates the dialogue state.

[0034] When subsequent voice input contains semantic references or omissions, this module completes the input using a rule-based engine, with rules executed according to priority: Location word matching: If the current input contains "left", "right", etc., then look up the most recent record in the cache whose body_part field contains the location word and use it to complete the current input; Fuzzy referential matching: If the current input contains "that", "it", etc., return the most recent record in the cache; Ellipsis completion: If the current input is an elliptical sentence such as "What about size?", then find the most recent record in the cache whose indicator field matches the current scenario (such as "follicle size"), and reuse its body_part and indicator; Pure numeric inheritance: If the current input is only a number (such as "18"), and the indicator entered in the previous round is "follicle size", then it will be automatically combined into {body_part: previous round body_part, indicator: follicle size, value: 18, unit: mm}.

[0035] When the memory function switch is false, no contextual memory is enabled, and each voice input is processed independently.

[0036] The User Interface (GUI) module provides a visual interface for doctors to interact with the system, enabling operation control, status display, result preview, and log recording. It consists of: a voice wake-up status indicator, a voice input start / stop button, a real-time voice transcription display area, a function switch configuration panel (memory / model switching / calibration), an operation result preview area, and a system operation log area. Operating environment: compatible with Windows 10 / 11 operating systems, supporting touchscreen, mouse, and keyboard operation.

[0037] The core configuration module centrally manages all configurable behavior switches in the system, serving as the global status center. It is responsible for receiving preset and dynamic adjustment instructions from doctors, maintaining the real-time status of switch parameters, and synchronizing these parameters to various business modules (such as the adaptive learning module and text proofreading module) through a unified interface. The core configuration module does not execute any specific business logic; it only provides storage and distribution services for configuration parameters.

[0038] The switch parameters of the core configuration module (all of which support real-time adjustment and include default values) include: Memory function switch: Boolean, defaults to true, controls whether to enable dialogue state tracking and context reuse; Model switching switch: enumeration type, optional values ​​are auto / command / text, default is auto, which affects the decision logic of input type judgment; Proofreading function switch: Boolean, defaults to true, controls whether the text proofreading model is enabled.

[0039] The voice wake-up module operates in the low-power background, monitoring the microphone audio stream in real time, recognizing preset wake words, and triggering the system to enter the formal speech recognition process, achieving voice-based startup without manual intervention. The module employs a lightweight CNN / MCUNet keyword detection model (model size ≤ 500KB, CPU utilization < 1%), with an audio sampling rate of 16kHz and a frame length of 25ms. Through Mel-spectrum feature extraction and neural network inference, it calculates the matching probability between the audio and the wake word in real time, determining wake word trigger when the matching probability is ≥ 0.8 across multiple consecutive frames. After wake-up, it sends a start signal to the speech recognition module and transmits the subsequent audio stream, while simultaneously providing immediate feedback to the doctor through GUI indicator flashing / voice prompts (such as "Yes"). Furthermore, the voice wake-up module allows administrators to customize / change wake words via the GUI, and the system automatically updates the wake word feature template.

[0040] The speech recognition module receives the speech stream after wake-up, performs audio preprocessing and feature extraction, and then passes the speech features to the learning module for input type classification.

[0041] The speech recognition module includes: Voice Activity Detector (VAD): Silence segment removal is achieved based on energy threshold + zero crossover rate, with the silence removal threshold set to -30dB; Feature extractor: Extracts 80-dimensional Mel spectrograms, frame length 25ms, frame shift 10ms, and converts the speech stream into a standardized speech feature vector (512-dimensional floating-point array).

[0042] The instruction set model identifies system operation commands in ovulation induction cycle monitoring scenarios, outputs standardized command IDs and confidence levels, and enables rapid response to system operations. Model architecture: Conformer-CTC, 8M parameters, supports streaming recognition. Training foundation: Based on 500 hours of voice training for ovulation induction cycle monitoring scenarios, covering 200 commonly used clinical operation commands (save, cancel, page turning, page switching, etc.). Performance indicators: Command recognition accuracy ≥98%, average response latency ≤200ms. Anomaly handling: When the confidence level of the recognition result is <0.7, the system prompts the doctor for manual confirmation via the GUI.

[0043] This free-text model recognizes free-text medical records dictated by doctors regarding assisted reproductive technology (ART), outputs standardized text, and extracts structured medical entities to suit clinical diagnosis and treatment scenarios. It employs a MedicalBERT+CRF model architecture, where the CRF layer globally constrains the label sequence output by BERT, ensuring the legality of entity label transfers (e.g., prohibiting "location" from being directly followed by "unit"). The model has 110M parameters. Fine-tuning was performed on a 1000-hour ART corpus, derived from de-identified electronic medical records from multiple partner hospitals. This corpus includes records of follicle monitoring and hormone testing, and entities are manually labeled (location, indicator, value, unit). Customized optimizations were made for core medical entities such as follicle size, hormone levels, and endometrial thickness in the context of ovulation induction cycle monitoring. Performance metrics achieved a text recognition accuracy of ≥92% and an F1 score of ≥0.89 for medical entity extraction. The output is standardized text combined with key-value pairs of structured entities (e.g., location: right ovary, indicator: follicle size, value: 18, unit: mm).

[0044] The text proofreading module performs secondary processing on the outputs of the instruction set model and the free text model throughout the entire process, ensuring the accuracy, standardization, and logical consistency of the entered content. It adopts a BART-base model architecture with 140M parameters. It is trained on 2 million error correction pairs from assisted reproductive medical texts, generated through automatic construction (e.g., replacing "endometrium" with "inner membrane") combined with manual review. Processing includes medical terminology correction (e.g., "inner membrane thickness" → "endometrial thickness"), format standardization (e.g., "18 mm" → "18mm"), and numerical logic verification. Performance metrics achieve an error correction accuracy of ≥90% and a single-sentence processing speed of ≤50ms.

[0045] The interface operation module is connected to the interface learning module and the text proofreading module respectively. Based on the operation intention and the operation mapping library, a three-level positioning strategy is adopted to achieve accurate positioning of third-party system controls. It also simulates manual operation through the system-level interface to complete interface-free data entry and system control, ensuring the robustness and high success rate of control positioning.

[0046] The three-level positioning strategy includes precise ID matching, fuzzy text label matching, and coordinate region positioning. Precise ID matching takes priority; if that fails, fuzzy text label matching is used, and if that also fails, coordinate region positioning is employed. This step-by-step fallback ensures robust control positioning. Upon successful positioning, operating system automation interfaces such as Windows SendInput or UI Automation are invoked to simulate mouse movement, clicks, keyboard input, and focus switching, enabling interface-free data entry and system control. Operations are automatically retried upon failure (maximum of 3 times, with a 1-second interval).

[0047] Furthermore, the operational intent specifically includes: Data population: Mapping structured medical data to the corresponding record fields in the target system; System operations: including executing system commands such as save, page turning, and page switching; Information Supplement: If the recognition result is incomplete, the user interaction module will prompt the doctor to supplement relevant diagnosis and treatment information.

[0048] A method for recording a reproductive ovulation induction cycle based on a multi-level ASR (Automatic Reduction Syndrome) collaborative system includes an offline learning phase and an online usage phase. The offline learning phase is only performed when initially connecting to the target system, during system upgrades, or when the interface changes. The online usage phase is the routine clinical procedure. Details are as follows: The core objective of the offline learning phase is to complete the interface parsing, feature extraction, and operation mapping library construction of the target third-party medical system, laying the foundation for interface-free operation. This includes the following steps: A1: Export the interface files of the target third-party medical system through the interface learning module; When the target third-party medical system is running normally, the administrator starts the interface learning module; the module automatically identifies the currently active window, exports its HTML structure for web applications, and generates a 1920×1080 resolution PNG format interface snapshot for desktop applications.

[0049] A2: Parse the interface file and identify interactive controls in the interface; The interface learning module reads the HTML file / interface snapshot exported in step A1 and adopts a differentiated parsing strategy: web applications extract all interactive UI elements through the DOM parser; desktop applications identify the control types and positions in the interface through a pre-trained YOLOv5 model.

[0050] A3: Extract the core features of each control; For each control identified by the parsing, fine-grained core features are extracted and recorded and stored in a temporary database. The features include: control type (input box / button / checkbox, etc.), screen coordinates (top left corner x / y) and width and height, unique identifier (id / name / automationId), associated text label, and executable operation (click / input / select, etc.).

[0051] A4: Organize the extracted control features into a structured operation mapping library; The control features in the temporary database are organized according to the window-page-control hierarchy of the target system to generate a structured operation mapping library in JSON format, clarifying the correspondence between interface elements and operation instructions.

[0052] A5: Perform simulated positioning and operation tests on all controls in the operation mapping library; The system automatically calls the interface operation module to simulate the positioning and operation test of all controls in the operation mapping library to verify the positioning accuracy and operation execution effectiveness. After the verification is successful, the mapping library is officially activated and can be called during the online use phase. If the verification fails, it returns to step A3 to re-extract features.

[0053] The online usage phase involves the voice recording process during doctors' daily consultations. Based on the operation mapping library built in the offline phase, it realizes a fully intelligent process from voice wake-up to interface-free recording with third-party systems. Each step is interconnected, including the following steps: B1: Doctors log in to the system through the user interaction module, and the core configuration module loads and synchronizes personalized switch parameters to each execution unit; Doctors log in to the system through the user interface module (GUI), and the core configuration module loads the default switch parameters. Doctors can adjust the on / off status of the memory function, model switching, and proofreading function in real time according to their personal treatment habits. The configuration parameters are synchronized in real time to the execution units such as the adaptive learning module and the text proofreading module through the core configuration module.

[0054] B2: The system monitors wake words in low power and enters recognition mode after a successful match, capturing the voice stream in real time. The system uses a low-power, background-only voice wake-up module to monitor the microphone audio stream in real time. When the doctor says the preset wake-up word (such as "good pregnancy") and the matching probability of multiple consecutive frames is ≥0.8, the system is woken up and enters the formal voice recognition mode. The doctor directly dictates the diagnosis and treatment information / operation instructions, and the voice stream is transmitted to the voice recognition module in real time with a sampling rate of 16kHz, 16bit, and mono.

[0055] B3: The speech recognition module extracts features from the speech stream and passes them to the adaptive learning module for input type discrimination; The speech recognition module removes silence segments and extracts 80-dimensional spectral features from the received speech stream, then passes the standardized speech features to the adaptive learning module. Upon receiving the speech features, the adaptive learning module extracts the control text labels of the current page from the operation mapping library as semantic anchors, calculates the semantic distance between the speech content and command-type features, and sets a classification threshold of 0.8: if the matching probability between the speech content and command-type features is ≥0.8, it is directly classified as a command; if the matching probability with free text-type features is ≥0.8, it is directly classified as free text; if both are below 0.8, a comprehensive judgment process is initiated, combining context and historical operation patterns for a secondary decision. The adaptive learning module also prioritizes matching the command set in the operation mapping library corresponding to the currently focused page / control, improving classification accuracy.

[0056] The specific logic for determining the type classification: The speech feature vector V (512-dimensional) output by the speech recognition module is first matched with the instruction feature template library. The instruction feature template library contains the feature vector Ti corresponding to each standard instruction (such as "save" or "page turn"). These templates are obtained by extracting features and clustering a large number of instruction speech offline.

[0057] Calculate the cosine similarity between V and each Ti: simi=(V·Ti) / (‖V‖×‖Ti‖) The maximum value, simmax, is taken as the matching probability with the instruction class.

[0058] Meanwhile, V is input into a pre-trained free text classifier (a lightweight binary classification model based on support vector machine SVM), and the output is the probability Ptext that belongs to free text.

[0059] Judgment rules: If simmax ≥ 0.8, it is determined to be an instruction class, output type 0, and confidence level = simmax.

[0060] If simmax < 0.8 and Ptext ≥ 0.8, then it is determined to be free text, output type 1, confidence level = Ptext.

[0061] If both are below 0.8, then a comprehensive judgment is made: combining the types of the three most recent operations (if all three are commands, then it tends to be a command) and the current screen focus (if the focus is on the input box, then it tends to be free text), the final type is determined by weighted voting; if it still cannot be determined, output type 2 (cannot be determined), confidence level = 0.5.

[0062] B4: Based on the input type discrimination result, the system calls the corresponding model to complete speech recognition and outputs standardized results; Based on the type determination result in step B3, the system calls the corresponding model to complete speech recognition and outputs the standardized result: If it is a command type: call the command set model, parse it into a standardized command ID and confidence level. If the confidence level is <0.7, prompt the doctor for manual confirmation through the GUI. If it is a free text class: call the free text model, output standardized medical text, and extract structured medical entities such as follicle size and hormone level, while completing unit standardization processing (such as "millimeters" → "mm").

[0063] B5: When the memory function is enabled in the core configuration module, the context management module maintains the dialogue state and completes the referential resolution and information completion. If the memory function switch in the core configuration module is true, the context management module starts dialogue state tracking, maintains the context cache, and completes referential resolution and information completion processing: The medical entities identified and extracted in this instance are stored in the context cache (the cache capacity is 10 rounds, and only the entities from the most recent 10 rounds of dialogue are saved). Update the current conversation status (e.g., "Entering follicle size data"); If subsequent speech contains semantic references or omitted information, the system will automatically search for the most recent entity of the same type in the context cache and complete the semantics.

[0064] B6: The text proofreading module performs terminology correction, format standardization, and numerical logic verification on the recognition results, and outputs the final standardized and structured data; The system inputs the processing results of steps B4 / B5 into the text proofreading model to complete three levels of processing: medical terminology correction, description format standardization, and numerical and logical verification, and outputs the final standardized and structured data that can be entered.

[0065] B7: The system integrates the current recognition results, context information, and semantics of the operation mapping library to parse out three types of operation intentions; The system combines the current recognition results, contextual information (when the memory function is enabled), and the semantics of the operation mapping library to parse out the specific operation intent.

[0066] B8: The interface operation module retrieves controls from the operation mapping library through a three-level positioning strategy, calls the automation interface to execute operations, and supports retries; Based on the operation intent in step B7, the interface operation module retrieves the target control from the operation mapping library using a three-level positioning strategy. Upon successful positioning, it calls the operating system's automation interface to simulate manual operation, completing data filling or system command execution. If the operation fails, it automatically retryes (up to 3 times, with a 1-second interval).

[0067] B9: The user interaction module displays the operation results in real time, allowing doctors to check and correct them. The system records the correction behavior for subsequent optimization. After the operation is completed, the user interface module (GUI) displays the operation results in real time, including the filled medical data, the executed system commands, and the success / failure status of the operation. Doctors can quickly check the results, and if errors are found, they can be corrected through voice commands or manual operation. The system automatically records the correction behavior to provide samples for subsequent optimization of the adaptive learning model.

[0068] B10: The adaptive learning module records data pairs before and after correction, and achieves personalized optimization and closed-loop feedback through incremental updates; When a doctor corrects the system's type judgment result (e.g., correcting text misclassified as an instruction to free text) or recognized content manually or verbally, the adaptive learning module automatically records the data pairs before and after the correction (original input + doctor's corrected label / content). These data pairs are stored in a dedicated incremental learning cache. During off-peak hours (e.g., at night) or after reaching a preset threshold, the system can use this new data to incrementally update the instruction feature vector space or fine-tune the free text model, allowing the system to gradually adapt to the doctor's personalized language habits and achieve closed-loop optimization of adaptive learning.

[0069] To clearly illustrate the practical application effects of this invention, a voice recording of a clinical follicle monitoring ultrasound examination in assisted reproductive technology is used as a specific scenario, combined with... Figure 1 The flowchart fully illustrates the entire workflow of the system from interface learning to voice input. The interface learning phase has been completed in advance, the operation mapping library is in effect, and the core configuration module parameters are: memory function enabled, automatic model switching, and proofreading function enabled.

[0070] Phase 1: Preliminary Preparations The system learns the interface of the target reproductive center's electronic recording system and constructs an operation mapping library for the ultrasound examination record page, which can accurately locate core controls such as the "follicle size input box" and the "save record button". After logging into the system and confirming the core configuration parameters, the doctor begins the ultrasound examination.

[0071] Phase 2: Single Voice Input and Saving During the ultrasound scan, the doctor uttered the wake-up phrase "Good pregnancy, good pregnancy," and after the system was activated, the doctor continued to verbally state: "Right ovarian follicle 18 mm, save," and the voice stream was transmitted to the system in real time. The core configuration module synchronizes the parameters for "memory enabled, automatic modeling, and calibration enabled" to all modules; The learning module receives speech features and calls the operation mapping library to complete semantic prediction. The first part "right ovarian follicle 18 mm" is judged as free text (match probability 0.95), and the second part "save" is judged as a command (match probability 0.98). The free text model identifies standardized text and extracts entities (location: right ovary, index: follicle size, value: 18, unit: mm), while the instruction set model identifies the "save" instruction (ID=save_001, confidence level 0.99). When the memory function is activated, the system stores the follicle entity in the context cache, and the current dialogue status is updated to "Enter follicle size data"; The text proofreading model verifies the recognition results to confirm that there are no terminological errors and that the format is correct, so no correction is needed. The system interprets the operation intent as follows: First, fill the "Follicle Size Input Box" with 18mm, and then execute the "Save Record" operation; The interface operation module locates the follicle size input box (x=150, y=250) and the save button (x=200, y=300) by ID matching, simulates mouse click + keyboard input of "18", and then simulates clicking the save button; The GUI provided real-time feedback: "Saved: Right follicle size 18mm." After the doctor quickly checked, the examination continued.

[0072] Phase 3: Context reuse in multi-turn dialogues (core effect of memory function) After completing the examination of the right ovary, the doctor continued to examine the left ovary, verbally stating, "The left one? 16 millimeters." The system then processed the following: Voice input is classified as free text, and the memory function is activated. The system detected semantic reference to "What about the left side?", retrieved the most recent "follicle size" entity from the context cache, and automatically completed the semantics to "left ovarian follicle size"; The free text model identifies "left ovarian follicle 16 mm", extracts the corresponding structured entity, and updates the context cache; The interface operation module locates the left follicle size input box, simulates inputting "16", and the GUI feedback "Left follicle size 16mm has been entered"; The doctor verbally says "save," and the system quickly executes the save operation, completing the cycle record for this follicle monitoring.

[0073] Key parameters of this invention are summarized below: The specific number of parameters can be scaled and adjusted according to hardware resource configuration and device / server deployment requirements.

[0074] Summary table of key parameters: .

[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / button>

Claims

1. A reproductive ovulation induction cycle recording device based on multi-level ASR coordination, characterized in that, It includes an interface learning module, an adaptive learning module, a context management module, a user interaction module, a core configuration module, a voice wake-up module, a speech recognition module, an instruction set model, a free text model, a text proofreading module, and an interface operation module; The interface learning module is used to connect with the target third-party medical system. It extracts control features by parsing the interface files and builds an operation mapping library. The adaptive learning module is connected to the speech recognition module and the interface learning module respectively. It dynamically judges the matching of speech input type, generates incremental training data by manually correcting behavior, and processes the input type. The context management module is connected to the adaptive learning module and the free text model respectively, and is controlled by the core configuration module. It is used to selectively enable context memory according to the preset memory function on / off state. The user interaction module provides a visual interaction interface between doctors and the system, enabling operation control, status display, result preview, and log recording. The core configuration module is used to centrally manage all configuration behavior switches in the system. It is responsible for receiving preset and dynamic adjustment instructions from doctors, maintaining the real-time status of switch parameters, and synchronizing the parameters to various business modules through a unified interface. The voice wake-up module is used to monitor the microphone audio stream in real time, recognize the preset wake-up word, and trigger the system to enter the formal voice recognition process. The speech recognition module is used to receive the speech stream from the voice wake-up module and complete audio preprocessing and speech feature extraction; The instruction set model is used to parse operation instructions and output standardized instruction IDs and confidence scores when the adaptive learning module determines that the input is an instruction class. The free text model is used to identify and output standardized text and extract structured medical entities when the adaptive learning module determines that the input is free text. The text proofreading module is used to perform secondary processing on the output of the instruction set model or free text model throughout the entire process. The interface operation module is connected to the interface learning module and the text proofreading module respectively. Based on the operation intention and the operation mapping library, it adopts a three-level positioning strategy to achieve accurate positioning of third-party medical system controls, and simulates manual operation through the system-level interface to complete interface-free data entry and system control.

2. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The interface learning module includes: The export unit is used to export the HTML file of the target third-party medical system or generate a high-resolution interface snapshot. The generation unit is used to export the interface file and save it in a standard format that the interface learning module can parse. The parsing unit is used to read and parse standard format interface files to complete the identification of interface structure and elements; The feature extraction unit is used to extract the control type, screen coordinates, unique identifier, associated text label, and executable operation; The mapping library building unit is used to organize the extracted control features into a structured operation mapping library in JSON format, establishing a one-to-one correspondence between interface elements and system operation commands.

3. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The adaptive learning module includes input type matching, human-computer interaction closed-loop optimization, and personalized model fine-tuning. Input type matching: Receives speech features from the speech recognition module, combines them with the operation mapping library of the interface learning module, and dynamically determines whether the speech input type is command type or free text type; Human-computer interaction closed-loop optimization: Record doctors' manual correction behavior to the system output, generate incremental training data, and provide samples for personalized model fine-tuning; Personalized model fine-tuning: Utilize the correction data accumulated through closed-loop optimization of human-computer interaction to periodically perform incremental updates on the feature vector space of the instruction set model or lightweight fine-tuning on the free text model.

4. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 3, characterized in that, Input type is determined by calculating the semantic distance between the speech content and the instruction class features. The type classification threshold is set to 0.8, and the specific determination is as follows: If the probability of matching the speech content with the command-type feature is ≥0.8, it is directly determined to be a command-type feature; If the matching probability with the free text category feature is ≥0.8, it is directly determined to be a free text category; If both are below 0.8, a comprehensive judgment process is initiated, and a secondary decision is made by combining the context and historical operating patterns.

5. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The context management module includes context memory management and reference resolution and information completion. When context memory is enabled: Context memory management: The system automatically maintains a dialogue state cache, recording the most recently identified ovulation induction monitoring medical entities and dialogue states; Substitution resolution and information completion: The recognition results output by the free text model are processed, semantic completion is performed on speech with semantic reference or semantic reference, and the completed semantic information is passed to the interface operation module.

6. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 5, characterized in that, The aforementioned referential resolution and information completion specifically include: If the current input contains a directional word, then find the most recent record containing that directional word in the context cache and complete it. If the current input contains a vague pronoun, return the most recent record from the context cache; If the current input is an elliptical sentence, then the missing numeric field is detected, and the most recent record matching the current scenario is searched from the context cache to complete it; If the current input is only a numerical value, and the metrics entered in the previous round match the current scenario, they will be automatically combined into a complete structured entity.

7. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The parameter switch includes: Memory function switch: Boolean, defaults to true, controls whether to enable dialogue state tracking and context reuse; Model switching switch: enumeration type, optional values ​​are auto / command / text, default is auto, which affects the decision logic of input type judgment; Proofreading function switch: Boolean, defaults to true, controls whether the text proofreading model is enabled.

8. The reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The operational intent specifically includes: Data population: Mapping structured medical data to the corresponding record fields in the target system; System operations: including executing system commands such as save, page turning, and page switching; Information Supplement: If the recognition result is incomplete, the user interaction module will prompt the doctor to supplement relevant diagnosis and treatment information.

9. A reproductive ovulation induction cycle recording device based on multi-level ASR coordination according to claim 1, characterized in that, The three-level positioning strategy, in order of priority, is ID exact matching, text label fuzzy matching, and coordinate region positioning. It calls the Windows SendInput / UIAutomation operating system automation interface to simulate mouse movement, clicks, keyboard input, and focus switching.

10. A method for recording ovulation induction cycles based on multi-level ASR coordination, applied to the ovulation induction cycle recording device based on multi-level ASR coordination as described in any one of claims 1 to 9, characterized in that, Includes offline learning phase and online usage phase: The offline learning phase includes the following steps: A1: Export the interface files of the target third-party medical system through the interface learning module; A2: Parse the interface file and identify interactive controls in the interface; A3: Extract the core features of each control; A4: Organize the extracted control features into a structured operation mapping library; A5: Perform simulated positioning and operation tests on all controls in the operation mapping library; The online usage phase includes the following steps: B1: Doctors log in to the system through the user interaction module, and the core configuration module loads and synchronizes personalized switch parameters to each execution unit; B2: The system uses low-power listening for wake words. After a successful match, it enters recognition mode and collects the voice stream in real time. B3: The speech recognition module extracts features from the speech stream and passes them to the adaptive learning module for input type discrimination; B4: Based on the input type discrimination result, the system calls the corresponding model to complete speech recognition and outputs standardized results; B5: When the memory function is enabled in the core configuration module, the context management module maintains the dialogue state and completes the referential resolution and information completion. B6: The text proofreading module performs terminology correction, format standardization, and numerical logic verification on the recognition results, and outputs the final standardized and structured data; B7: The system integrates the current recognition results, context information, and semantics of the operation mapping library to parse out three types of operation intentions; B8: The interface operation module retrieves controls from the operation mapping library through a three-level positioning strategy, calls the automation interface to execute operations, and supports retries; B9: The user interaction module displays the operation results in real time, allowing doctors to check and correct them. The system records the correction behavior for subsequent optimization. B10: The adaptive learning module records data pairs before and after correction, and achieves personalized optimization and closed-loop feedback through incremental updates.

Images