AR smart glasses multi-modal patrol data processing method and system

Abstract

The application relates to the technical field of medical informatization, and discloses an AR intelligent glasses multi-modal ward round data processing method and system, which comprises a recognition patient identity and peripheral equipment binding state regulation data writing window; structured medical record slots are generated by synchronously acquiring dialogue voice, affected part images and instrument numerical values, and aggregating the numerical values according to a time window; multi-source confidence weighted fusion calculation is performed, and interactive signal confirmation cover machine numerical values are captured; comparison and standard template generation generate follow-up questions or supplementary shooting tasks, the state of an operator is analyzed, the task execution queue is dispatched, visual rendering output is issued, an edge module is called to execute mask desensitization, and then evidence-based retrieval is initiated and a structured archiving file is generated. The application improves the accuracy of medical record filing by multi-source data confidence fusion and artificial short circuit confirmation, guarantees the continuity of diagnosis and treatment based on personnel state queue scheduling prompt, and combines end-side desensitization and dynamic degradation retrieval to give consideration to data privacy security and search timeliness.

Description

Technical Field

[0001] This application relates to the field of medical information technology, specifically to a method and system for processing multimodal mobile medical data for AR smart glasses. Background Technology

[0002] With the development of medical informatization, mobile medical devices such as AR smart glasses are increasingly being used in ward rounds to assist medical staff in recording patient vital signs and medical records. However, current data collection methods for ward rounds are relatively simple and easily affected by environmental noise or visual obstruction in complex clinical environments, leading to errors. While some devices can acquire multiple signals, they lack effective data synchronization and confidence assessment mechanisms when processing heterogeneous data such as voice, images, and instrument interfaces. When data from different collection channels shows numerical discrepancies, existing systems struggle to determine data accuracy and lack short-circuit verification logic to introduce manual intervention when machine recognition confidence is low, resulting in low accuracy of the final entered and archived medical record data.

[0003] During clinical rounds, operators need to perform aseptic procedures or focus on current doctor-patient communication and physical examinations. Existing AR medical devices, when the system detects missing medical record fields and generates follow-up questions or supplementary data collection tasks, use instant pop-up windows to directly provide visual prompts on the glasses screen. This forced information push, indiscriminately pushing information regardless of the clinical scenario, can easily obstruct the operator's view, causing visual interference with ongoing critical clinical procedures in non-emergency situations, affecting the continuity and safety of diagnostic and treatment procedures.

[0004] To assist in rounds and diagnoses, the device needs to initiate retrieval requests from the medical knowledge base. Most existing retrieval solutions directly upload raw data containing patient identity and sensitive vital signs to the cloud for processing, which poses a risk of patient privacy leakage during data transmission and external processing. Furthermore, wireless networks in hospital wards and similar environments experience signal fluctuations or coverage blind spots, making complete reliance on cloud retrieval prone to request blocking under poor network conditions, compromising the timeliness of medical searches. The collected multimodal heterogeneous data also lacks an automated structural encapsulation mechanism based on logical hierarchical relationships, increasing the difficulty of subsequent data traceability, auditing, and archiving in the medical record system. Therefore, a new technical solution is needed to address these issues. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal mobile medical data processing method and system for AR smart glasses. This solves the problems of existing mobile medical equipment being prone to errors in multi-source data acquisition, resulting in low accuracy of medical record recording; forced pop-up prompts interfering with the continuity of clinical operations; and direct cloud retrieval posing a risk of patient privacy leakage and poor retrieval timeliness during network fluctuations.

[0006] To achieve the above objectives, this application provides the following technical solution: The first aspect of this application provides a method for processing multimodal mobile medical data from AR smart glasses, including the following steps: The system identifies the patient's identity and generates identity recognition status data; it establishes a binding connection with the external detection equipment and generates binding status data; and it adjusts the data writing window to be open or suspended based on the identity recognition status data and the binding status data. While the data writing window is open, the system synchronously acquires dialogue voice, images of the affected area, and instrument values. It then performs aggregation processing on the heterogeneous data according to the time window segmentation standard to generate structured medical record slots carrying source reference parameters. Perform multi-source confidence weighted fusion calculation on multiple candidate values ​​in the structured medical record slots and output conflict slot records. Capture the interaction signals triggered by the conflict slot records and execute short-circuit confirmation logic to overwrite the machine-calculated values. The system reads the department's standard template and performs a set comparison operation with the acquired structured medical record slots to generate follow-up questions or reshoot tasks. It analyzes the environmental sound field characteristics and pose sequence to determine the operator's status, and performs queuing scheduling for the follow-up questions or reshoot tasks to extract text prompt fields and send them for rendering output. The edge AI acceleration module is invoked to perform masking operations to generate de-identified query request data, initiate evidence-based medicine retrieval, insert medical advice citation information into the corresponding structured medical record slots, and summarize to generate structured archive files.

[0007] Furthermore, adjusting the data writing window to be open or suspended based on the identity recognition status data and the binding status data includes: Visual and spatial positioning features are extracted to identify the patient's identity and generate the identity status data. The low-power Bluetooth broadcast packets sent by the bedside peripheral device are scanned and combined with the patient's bed identification to verify and establish a binding connection relationship to generate the binding status data. When the identity matching score is greater than or equal to the matching score threshold and the device binding status parameter is in the bound state, the data writing window status parameter is changed to the open state. If the duration of no effective doctor-patient interaction is detected exceeds the silent duration threshold, the data writing window state parameter is changed to a suspended state, and the corresponding subsequent data is transferred to the isolation buffer for temporary storage.

[0008] Furthermore, the generation of the structured medical record slot carrying the source reference parameter includes: Simultaneously acquire the dialogue voice, affected area image and instrument values, and bind them with physical time and spatial positioning parameters to generate a heterogeneous data acquisition package; Extract heterogeneous data collection packets that fall within the same time window interval, perform parsing operations, and map them to medical record field names; Instantiate a structured medical record slot containing candidate values, timestamps, and source reference parameters, and store the globally unique record identifier within the same time window into the multimodal source reference parameter set.

[0009] Furthermore, the step of performing multi-source confidence weighted fusion calculation and outputting conflict slot records, and capturing the interaction signal triggered by the conflict slot records to execute short-circuit confirmation logic, includes: The reliability index of independent data sources, the output score of the algorithm model, and the time distance weight parameter are extracted to perform multi-source confidence weighted fusion calculation to obtain the slot fusion confidence score; When the absolute value of the difference between multiple candidate values ​​is greater than the numerical conflict tolerance threshold, there is a cross-modal logic contradiction, or the slot fusion confidence score is lower than the fusion confidence threshold, the conflict slot record with the pending conflict mark is output. Based on the captured interaction signal parsing, a confirmation action is generated, and short-circuit confirmation logic is executed according to the confirmation action to force the verification result to overwrite the machine-calculated value.

[0010] Furthermore, the generation of follow-up questions or reshoots includes: Extract routine clinical features from the department's standardized template and perform set comparison operations with the confirmed structured medical record slots to obtain the set of gaps in the rounds data; For the missing fields in the aforementioned outpatient data gap set, the action type is determined to be either a follow-up questioning task or a reshooting task. A priority value is calculated by combining the medical risk weight and the duration of the missing data, and this value is used to generate the follow-up questioning or reshooting task.

[0011] Furthermore, the process of determining the operator's status and performing queuing scheduling for the follow-up questioning or reshooting task, extracting text prompt fields, and sending them for rendering output includes: The environmental sound field characteristics and the pose sequence of the inertial sensor are analyzed, and the status identifier records are output, which are divided into voice communication state, aseptic operation state, or idle state. When the current state is either voice communication or aseptic operation and the priority value is less than the critical priority threshold, the follow-up questioning or reshooting task will be kept in the waiting queue to postpone the visual prompting time. When the current state is idle or the priority value is greater than or equal to the critical priority threshold, the text prompt field of the first ranked follow-up question or reshoot task is extracted and sent to the optical waveguide AR display module for visual rendering output.

[0012] Furthermore, initiating an evidence-based medicine search includes: The NPU edge AI acceleration module is invoked to locate the area containing identity privacy information, perform a masking operation, and generate the de-identified query request data; Extract medical keywords from the de-identified query request data and perform vector similarity calculation with the local offline knowledge base index to obtain the local offline knowledge base hit rate; When the hit rate of the local offline knowledge base is less than the hit rate threshold and the network latency parameter is less than the network latency threshold, a cloud-based evidence-based medicine data retrieval is initiated; otherwise, a local evidence-based medicine data retrieval is initiated.

[0013] A second aspect of this application provides an AR smart glasses multimodal mobile medical data processing system, applied to the method provided above, including: The patient bed identification module, the vital signs equipment dynamic binding module, and the round-trip session management module are used to identify patient identity identifiers to generate identity identification status data, establish binding connection relationships with peripheral detection equipment to generate binding status data, and adjust the data writing window to be open or suspended based on the identity identification status data and the binding status data. The multimodal acquisition module and slot generation module are used to simultaneously acquire dialogue voice, affected area images and instrument values ​​when the system is enabled. They also perform aggregation processing on heterogeneous data according to the time window segmentation standard to generate structured medical record slots carrying source reference parameters. The confidence fusion and conflict detection module and the doctor confirmation module are used to perform multi-source confidence weighted fusion calculation on candidate values ​​in the structured medical record slots and output conflict slot records, capture interaction signals and execute short-circuit confirmation logic to overwrite machine-calculated values; The outpatient gap detection module, doctor status recognition module, and AR prompt module are used to read the department standard template and the acquired structured medical record slots to perform set comparison operations to generate follow-up questions or reshoot tasks, analyze the environmental sound field features and pose sequence to determine the operator's status, and perform queuing scheduling for the follow-up questions or reshoot tasks to extract text prompt fields and send them for rendering output. The edge-side de-identification module, de-identification RAG routing module, and audit archiving module are used to call the edge AI acceleration module to perform masking operations to generate de-identified query request data, initiate evidence-based medicine retrieval, insert medical advice citation information into the corresponding structured medical record slots, and summarize to generate structured archive files.

[0014] Furthermore, the system also includes a physical hardware architecture built on AR smart glasses, which includes a main control SoC, an NPU edge AI acceleration module, an input link, and an output link. The input link includes a first-view camera module, a microphone array, an inertial sensor, a wear detection unit, and a vital signs interface; The output link includes an optical waveguide AR display module and a bone conduction speaker; The multimodal acquisition module is used to activate the input link to concurrently capture environmental audio, first-view images, and peripheral vital sign interface data. Based on a globally unified hardware clock system, it generates standardized data acquisition packages by binding physical time and spatial positioning parameters to heterogeneous data.

[0015] Furthermore, the specific logic for the audit archiving module to generate structured archive files is as follows: Perform hierarchical nesting processing according to logical hierarchy, set the session identifier and operator identifier of the current round of visits as the root node of the data tree, and set the confirmed structured medical record slot as the branch node; The physical storage pointers of multimodal data sources, manual review logs, and evidence-based traceability links are bound to the leaf nodes of the corresponding branch nodes. The memory objects that present a tree structure are serialized, merged, and packaged into a structured archive file with dimensional association attributes.

[0016] This application provides a method and system for processing multimodal mobile medical data for AR smart glasses, which has the following beneficial effects: 1. This application synchronously acquires dialogue voice, affected area images and instrument values ​​and aggregates them according to time windows. It combines the reliability index of independent data sources, model scores and time weight parameters to perform multi-source confidence weighted fusion calculation. At the same time, when there is a difference conflict in the candidate values ​​or the confidence is too low, it captures the interaction signal and executes the forced confirmation logic to overwrite the machine values. This solves the problem that a single acquisition method is prone to introducing errors and improves the accuracy of medical record data acquisition and filing.

[0017] 2. This application determines the current clinical action status of the operator by analyzing the environmental sound field characteristics and pose sequence, and performs queuing scheduling by comparing the priority values ​​of follow-up questions or reshoot tasks generated by the department template. When the personnel are in an aseptic operation state and the task has not reached the critical priority, the prompt will be retained in the waiting queue and output later, avoiding visual interference from non-urgent visual information to the critical clinical operation and ensuring the continuity of diagnosis and treatment.

[0018] 3. This application calls the edge computing module to locate the area containing privacy information, performs masking operations to generate de-identified query request data, and dynamically performs cloud retrieval or downgrades to initiate local retrieval based on the local knowledge base hit rate and network latency parameters. This can maintain the timeliness of retrieval in unstable network environments while preventing the leakage of sensitive patient information, and realize the automatic structural encapsulation and archiving of heterogeneous data according to the subordinate relationship. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0022] Figure 1 This is an architecture diagram of the AR smart glasses multimodal mobile clinic data processing system of this application; Figure 2 This is a flowchart of the multimodal mobile clinic data processing method for AR smart glasses in this application; Figure 3 This is a flowchart illustrating the patient identification and writing window control process for this application. Figure 4 This is a flowchart of the multimodal data acquisition and slot generation process in this application; Figure 5 This is a flowchart of the conflict detection and doctor's confirmation process for this application; Figure 6 This is a flowchart of the gap detection and intelligent prompting scheduling in this application; Figure 7 This is a flowchart of the de-identification retrieval and audit archiving process for this application; Figure 8 This is a comparison chart of multi-source confidence scores in this application; Figure 9 This is a diagram showing the relationship between the priority and duration of the gap tasks in this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0025] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of one element or feature relative to another element or feature, as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "below," "above," "front," "back," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure. For example, if the device in the figure undergoes a positional flip, orientation change, or change of motion, these directional indications will change accordingly. For instance, an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented "above other elements or features" or "above other elements or features." Therefore, the example term "below" can include both upper and lower orientations. The device may be otherwise oriented (rotated 90 degrees or in other directions), and the spatial relative descriptors used in the text will be interpreted accordingly.

[0026] See attached document Figure 1 This application provides a multimodal mobile medical data processing system for AR smart glasses. The system's physical hardware architecture is built upon the AR smart glasses themselves. The AR smart glasses internally configure a main control SoC and an NPU edge AI acceleration module to perform local edge-side data preprocessing and computational inference. The input links of the AR smart glasses include a first-person view camera module, a microphone array, an inertial sensor, a wear detection unit, and a vital signs interface. The vital signs interface supports Bluetooth Low Energy access and wired connection. The output links of the AR smart glasses include an optical waveguide AR display module and a bone conduction speaker. The main control SoC establishes electrical connections with the wireless communication module and the battery power management unit, respectively. The system relies on the physical hardware to run data processing logic, and the software architecture is divided into multiple collaborative modules, including: The patient bed identification module receives image data and environmental features acquired by the first-view camera module, extracts features of the bedside QR code, wristband code, and ward spatial positioning features, executes a multimodal cross-validation algorithm, and outputs identification status data containing patient identity identifiers and matching scores.

[0027] The outpatient session management module receives patient identification, instantiates a dedicated outpatient session object for the current patient in memory, and adjusts the open, suspended, or closed state of the data writing window based on the identification status data and location change parameters to construct an anti-crosstalk isolation zone for the current patient's data.

[0028] The vital signs device dynamic binding module is used to scan near-field communication tags or parse peripheral low-power Bluetooth broadcast packets, extract signal strength parameters and device identifiers, compare them with the hospital's equipment ledger to verify device ownership, output binding status data representing the current connection relationship of peripheral detection devices around the bed, and synchronize the binding status data to the round-trip session management module to assist in the execution of window control.

[0029] The multimodal acquisition module is used to simultaneously acquire dialogue voice recorded by the microphone array, images of the affected area and examination reports acquired by the first-person view camera module, and instrument values ​​returned by the vital signs interface when the data writing window is open. When the writing window is suspended, subsequent data acquired by the multimodal acquisition module is transferred to the isolation buffer for temporary storage.

[0030] The slot generation module reads the physical time coordinates of the input data from the multimodal acquisition module, performs aggregation processing on heterogeneous data according to the preset time window segmentation standard, maps and splices data elements belonging to the same time window and pointing to the same clinical significance, and generates structured medical record slots containing source reference parameters, timestamps and candidate values.

[0031] The confidence fusion and conflict detection module is used to parse multiple candidate values ​​contained in the structured medical record slots, extract data source reliability indicators, algorithm model output scores, and time distance weight parameters, and perform multi-source confidence weighted fusion calculations. Based on set threshold rules, the module determines the status of data items and outputs conflict slot records with pending conflict markers for low-confidence candidate values ​​or cross-modal logical contradictions.

[0032] The round-call gap detection module is used to read the preset department standard template and the structured medical record slots already acquired in the current session, perform set comparison operation, calculate the data completeness parameters of the current round-call task, extract missing items, and generate follow-up questions or reshoot tasks that include action type, text prompt fields, and priority.

[0033] The doctor status recognition module is used to analyze the environmental sound field characteristics collected by the microphone array and the pose sequence of the inertial sensor, perform voice activity detection and posture recognition calculation, and determine and output an identifier record indicating whether the current operator is in a voice communication state or a sterile operation state.

[0034] The AR prompt module is used to receive follow-up questions or retake tasks, read the identifier records output by the doctor's status recognition module to perform queuing and scheduling, postpone the prompting time for non-critical tasks, and extract the text prompt field of the first task in the queue and send it to the optical waveguide AR display module for visual rendering output.

[0035] The doctor confirmation module is used to capture voice commands, key signals, or gesture interaction signals triggered by operators in response to conflict slot records, parse the review results given by the operators, execute short-circuit confirmation logic, and force the review results to overwrite the machine-calculated values ​​output by the confidence fusion and conflict detection modules.

[0036] The edge-side de-identification module is used to call the NPU edge AI acceleration module to perform masking operations on the retained images and text, remove identity-sensitive entities, and generate de-identified summary data and de-identified query request data.

[0037] The de-identified RAG routing module is used to receive de-identified query request data, evaluate the hit rate of the local offline knowledge base and the current network status, execute the end-to-cloud adaptive routing strategy to initiate evidence-based medicine data retrieval, parse the returned medical advice citation information and insert it as evidence links into the corresponding structured medical record slots.

[0038] The audit archiving module is used to summarize and merge confirmed structured medical record slots, manual review logs, and evidence-based traceability links to generate a structured archive file for the current round of visits with dimensional correlation attributes.

[0039] See attached document Figure 2 This application provides a method for processing multimodal mobile medical data from AR smart glasses, including the following steps: S100, the patient bed identification module extracts visual features and spatial positioning features to calculate the current patient's identity, the vital signs equipment dynamic binding module establishes the binding connection relationship of the current bed's peripheral detection equipment, and the rounds session management module adjusts the opening and hanging status of the data writing window based on the identity recognition status data and binding status data. S200: The multimodal acquisition module simultaneously acquires dialogue voice, affected area images and instrument values ​​when the data writing window is open. The slot generation module performs aggregation processing on heterogeneous data according to the time window segmentation standard to generate structured medical record slots carrying source reference parameters. S300, the confidence fusion and conflict detection module extracts multiple candidate values ​​in the structured medical record slots, performs multi-source confidence weighted fusion calculation, and outputs conflict slot records. The doctor confirmation module captures the interaction signal triggered by the operator, performs short-circuit confirmation logic to overwrite the machine-calculated value. S400, the round-call gap detection module reads the department standard template and the acquired structured medical record slots to perform set comparison calculations to generate follow-up questions or reshoot tasks; the doctor status recognition module analyzes the environmental sound field features and pose sequence to determine the operator's status; the AR prompt module performs queuing scheduling for tasks based on the judgment results, extracts text prompt fields, and sends them down for rendering output. In the S500, the edge-side desensitization module calls the edge AI acceleration module to perform masking operations and generate desensitized query request data. The desensitized RAG routing module initiates evidence-based medicine retrieval and inserts the returned medical advice reference information into the corresponding structured medical record slot. The audit archiving module summarizes the associated records and generates structured archive files.

[0040] To enable those skilled in the art to better understand this application, the following will elaborate on the technical details and specific implementation logic of the above-mentioned system operation and method execution process.

[0041] See attached document Figure 3 The process of extracting visual and spatial positioning features to calculate patient identification and establishing device binding connection in step S100 is specifically implemented through the following steps: S101, the patient bed identification module extracts visual features and spatial positioning features to calculate the current patient's identity identifier, and triggers the rounds session management module to instantiate the rounds session object.

[0042] When doctors wear augmented reality (AR) smart glasses to perform mobile rounds in the ward, the first-person view camera module on the AR smart glasses acquires image data of the current bed area. The patient bed identification module receives the image data and uses a pre-built machine vision algorithm to extract features, parsing out visual features including a QR code at the head of the bed, the patient's wristband code, and the ward door number. Simultaneously, the patient bed identification module obtains the current spatial positioning features of the ward by receiving wireless beacon signals deployed within the ward. For the specific implementation of extracting QR code features from the image data or parsing wireless beacon signals to obtain positioning features, those skilled in the art can use open-source computer vision libraries and conventional indoor positioning algorithms; the implementation principles are well-known in the field and will not be elaborated here.

[0043] After acquiring the aforementioned multidimensional features, the patient bed identification module performs cross-validation calculations based on these features. When the identified bedside QR code parsing information and wristband code parsing information point to the same medical record, or when visual features match spatial positioning features, identity recognition status data containing the patient's identification identifier and matching score is generated. The round-trip session management module receives the patient's identification identifier and instantiates a dedicated round-trip session object corresponding to the current patient in the system's memory space. This object is defined as a multi-attribute data set, expressed by the formula: ; in, Indicates the object of the rounds session; Indicates the global round identifier assigned by the system; This indicates the patient's identity as confirmed through calculation; Indicates the location of the patient's bed; The identifier of the doctor performing the procedure; This indicates the preset clinical standard template identifier for the corresponding department; and These represent the physical times of session establishment and termination, respectively; The parameter represents the lock status of the session and is used to control whether multimodal acquisition data is allowed to be written to the current data record in subsequent processes.

[0044] S102, the vital signs equipment dynamic binding module scans peripheral signals to extract feature parameters, verifies equipment ownership by combining with the hospital equipment ledger, and establishes the binding connection relationship of peripheral detection equipment around the bed.

[0045] In mobile medical consultation scenarios, adjacent beds in the ward are equipped with medical devices that support wireless signal transmission, such as monitors and infusion pumps. To prevent augmented reality smart glasses from mistakenly connecting to vital sign interface devices on neighboring beds, the vital sign device dynamic binding module uses its built-in wireless communication module to scan low-power Bluetooth broadcast packets sent by peripheral devices near the bed, or read near-field communication tags attached to the devices, to extract the signal strength parameters of the wireless signal and the physical machine identifier of the device.

[0046] The vital signs equipment dynamic binding module combines the acquired physical machine identifier with the aforementioned hospital bed identifier. The system calls the equipment ledger interface provided by the hospital's internal information system to verify whether the current physical machine identifier belongs to the identified patient bed. After the ledger attribution verification is successful, the vital signs equipment dynamic binding module further evaluates the physical distance characteristics. Distance verification is considered successful when the received near-field communication tag contact result is valid, or when the Bluetooth Low Energy signal strength parameter is greater than the preset near-field signal threshold. The near-field signal threshold is set by those skilled in the art to be between -60dBm and -45dBm, based on the physical space span of a typical single-patient bed, to ensure that the accessed signal source is within the current effective physical radius.

[0047] After successful verification, the vital signs device dynamic binding module synchronously generates binding status data in memory, representing the current connection relationship of peripheral monitoring devices around the hospital bed. The data structure is expressed by the following formula: ; in, Indicates the binding status data; and These respectively represent the unique identifier and type of the peripheral device; Indicates the source of the binding; and These represent the measured signal strength parameters and near-field communication tag data, respectively. This indicates the characteristics of the device barcode obtained through scanning; Indicates the device binding status parameters; Indicates the physical moment when a valid signal was last detected; This indicates the set connection timeout period.

[0048] After the object is generated, the vital signs device dynamic binding module will... The value is changed to a bound state, establishing a legitimate data access link for the current bed. After the binding connection is established, the augmented reality smart glasses can accurately receive the instrument values ​​returned by the corresponding detection equipment through the vital signs interface, thus preventing and isolating cross-bed data crosstalk.

[0049] After completing the patient identification calculation and establishing the device binding connection, the system achieves cross-bed data crosstalk prevention and isolation by dynamically adjusting the data writing window. This is achieved through the following steps: S103, the outpatient consultation session management module controls the data writing window to be open based on the identity recognition status data and binding status data.

[0050] After acquiring the identity verification status data and binding status data generated in the aforementioned steps, the outpatient session management module integrates these dimensions of status information in the background and performs global control over the write window. The system summarizes the current identity cross-verification results and the legitimate access status of the device, and determines whether to grant data write permissions for the current outpatient session by conditionally judging the internal parameters of the relevant status. The data structure of this globally controlled status set is expressed by the formula: ; in, Represents the set of states used for global control; Indicates the global round identifier assigned by the system; This represents the set of identification sources used in the identity verification process; This represents the identity matching score calculated by integrating multimodal features; This indicates the currently active and bound patient identifier; The data write-in window status parameter indicates the direction of data flow control. This represents an identity conflict flag used to indicate whether identity information is contradictory; This indicates the device binding status parameters established by the aforementioned steps.

[0051] After generating this state object, the clinic session management module continuously monitors the values ​​of various parameter variables. When the identity matching score... Greater than or equal to the set matching score threshold And identity conflict identifier The device binding status parameters of the associated bed are in a conflict-free state. When the system is in a bound state or confirms that no vital signs device is connected, the round-trip session management module will write data to the window status parameters. The value has been changed to the enabled state. Matching score threshold The value range is set by those skilled in the art to be between 0.85 and 0.95 based on the confidence distribution of visual algorithms. While the data writing window is open, all speech, image, and vital sign data acquired by the multimodal acquisition module are allowed to be directly written to the currently assigned global round identifier. In the corresponding session logs.

[0052] S104, the outpatient session management module monitors environmental and status changes, writes data to the window and suspends it, and transfers the collected data within the corresponding time period to the isolation buffer.

[0053] During continuous rounds, doctors are constantly on the move and may shift their gaze or change the object of their care. The rounds session management module monitors the doctor's gaze area and device signal changes in real time through the input link of the augmented reality smart glasses. When the system detects inconsistencies between the wristband's parsing characteristics and the bedside code's parsing characteristics, discrepancies between the voice address captured by the system and the name recorded in the current medical record, failure to match the device's unique identifier with the patient's bed identifier, or when the signal strength parameters of peripheral devices are lower than the set near-field signal threshold or the set connection timeout period has passed without receiving the device's heartbeat data packet, the system determines that the current operating environment has changed.

[0054] The outpatient consultation session management module has a set silent duration threshold. The silence duration threshold is set to 30 to 60 seconds. When the system fails to detect effective doctor-patient interaction for a duration exceeding this silence duration threshold, or when the doctor's gaze deviates from the current bed area, the rounds session management module forcibly writes the data into the window status parameters. The value of is changed to a suspended state. For the specific implementation of determining gaze deviation or estimating gaze point by analyzing first-person view camera images, those skilled in the art can use eye-tracking algorithms or head pose estimation algorithms combined with scene semantic segmentation. The specific calculation logic is well-known in the field and will not be elaborated here.

[0055] When the data writing window is suspended, the multimodal acquisition module does not stop working. Instead, it transfers the subsequent data streams to the system's isolation buffer for temporary storage. Within the isolation buffer, the continuously acquired data stream is divided into multiple independent sample data objects, the data structure of which is expressed by the formula: ; in, This represents a sample data object within an isolated buffer. Indicates sample identification; Indicates the modality type corresponding to the data; Indicates the physical time of data sampling; This represents a raw data reference pointer that points to the underlying audio / video or vital sign data. This indicates the quality score of the data collection. This indicates the candidate round identifier predicted by the system.

[0056] Sample data objects temporarily stored in the isolation cache cannot be automatically incorporated into the formal electronic medical record; their lifecycle is controlled by a confirmation mechanism. Only after the system captures an attribution confirmation interaction triggered by a doctor is the sample data object approved for transfer to the corresponding rounds session. The rounds session management module has a set threshold for sample expiration time. The value range is set to 5 to 10 minutes. If the residence time of a sample data object in the isolation buffer exceeds the sample expiration time threshold and no confirmation instruction has been received, the system determines that the data has lost its validity and executes the expiration cleanup logic. During the cleanup process, the system retains the de-identified text digest extracted by the edge processing unit for log verification and discards the original data reference pointer. This includes the associated original image frames and audio files. This mandatory anti-crosstalk isolation zone design cuts off the physical channels for environmental noise and neighboring bed images to accidentally enter the current medical record.

[0057] See attached document Figure 4 With the data writing window set to the open state, the process in step S200 regarding the multimodal acquisition module simultaneously acquiring dialogue voice, affected area image, and instrument values ​​is specifically implemented through the following steps: S201, the multimodal acquisition module activates the system's multi-input link hardware to concurrently capture environmental audio, first-person view images, and peripheral vital sign interface data.

[0058] When the consultation session management module sets the data writing window status parameter to the open state, the multimodal acquisition module immediately activates the multi-input interface of the augmented reality smart glasses. In the audio acquisition link, the microphone array continuously records the ambient acoustic signals within the doctor's operating range. The system's underlying layer performs preliminary separation of the acoustic signals, extracting doctor-patient dialogue voice segments and the doctor's independently dictated medical instructions. For the sound source localization and background noise suppression processing of the multi-channel audio signals, those skilled in the art can use microphone beamforming and adaptive filtering algorithms. The algorithm logic is well-known in the field and will not be elaborated here.

[0059] In the visual acquisition chain, the first-person view camera module continuously captures the video stream in front of the doctor's field of vision at a set sampling frame rate. This sampling frame rate is set between 15 frames per second and 30 frames per second, and its specific value is dynamically determined by the power management unit inside the main control SoC in combination with the current battery level and computing load of the system, so as to control the power consumption of the end device while ensuring the smoothness of image capture. The multimodal acquisition module automatically adjusts the focus parameters of the camera based on the contrast feedback in the center of the image, capturing close-up images of the appearance of the patient's affected area and high-resolution images of paper examination forms in the medical record folder.

[0060] In the vital signs data access link, the multimodal acquisition module listens for and receives instrument values ​​transmitted back from external detection devices through the previously established vital signs interface. Since the binding status parameters of the target device have been set to "bound" in the preceding steps, the multimodal acquisition module only extracts instrument value data from this legitimate access link when parsing the underlying communication messages. The hardware driver layer directly discards and rejects broadcast data sent by unbound devices within the same frequency band, thereby ensuring the uniqueness and accuracy of the data source.

[0061] S202, the multimodal acquisition module binds unified physical time and spatial positioning parameters to heterogeneous data based on a global hardware clock, generating standardized data acquisition packages.

[0062] Because the physical sampling frequencies and internal transmission delays of the microphone array, camera module, and vital signs interface differ, the multimodal acquisition module maintains a globally unified hardware clock system at the kernel level of the main control SoC to ensure time base alignment for subsequent data aggregation and segmentation. Whenever the audio buffer fills a frame, the camera image sensor outputs a frame of pixel data, or the Bluetooth protocol stack throws an instrument value message, the system's hardware interrupt service routine immediately records the current physical moment of acquisition for that data.

[0063] The system reads the current pose data from the inertial sensor and the beacon base station signals in the area, generating a physical spatial positioning tag containing three-dimensional spatial coordinates and device orientation parameters. The multimodal acquisition module structures and encapsulates the original media stream segments or numerical strings with their corresponding time and spatial parameters, generating a heterogeneous data acquisition package. The structure of this heterogeneous data acquisition package is expressed by the formula: ; in, This represents the heterogeneous data acquisition package generated through encapsulation; This represents a globally unique record identifier assigned by the system to the current continuously acquired segment; Indicates the modality type of the data, including audio streams, video frames, or vital sign messages; This represents the actual underlying storage of the audio binary stream, image pixel matrix, or instrument numerical payload; Indicates the uniform physical moment of hardware interrupt allocation; This indicates the bound physical location tag.

[0064] After this encapsulation, the multimodal heterogeneous data is no longer an isolated and disordered streaming media segment on the timeline, but a standardized event object with high alignment accuracy. The multimodal acquisition module submits the generated multiple heterogeneous data acquisition packages to the main control computing bus, providing basic data support for subsequent stages to perform mapping and splicing of different modal data and slot extraction according to preset time window standards.

[0065] After generating a standardized data collection package, the system performs semantic-level aggregation on heterogeneous data using a time window strategy, thereby extracting and generating traceable structured medical record fields. This is achieved through the following steps: S203, the slot generation module merges the same source multimodal acquisition packages according to the preset time window segmentation standard, and maps them to specific medical record field names through the semantic model.

[0066] The slot generation module sets a time window segmentation standard for data extraction in the system's memory space. This time window segmentation standard includes a time window threshold. Time window threshold The value range is set by those skilled in the art to be between 3 and 5 seconds, based on the average time interval between a doctor's verbal statements and physical actions during routine clinical consultations. The slot generation module uses this time window threshold as a sliding step size to continuously compare the physical time of each heterogeneous data acquisition packet generated in the aforementioned steps with the session record corresponding to the currently active global round identifier.

[0067] When multiple heterogeneous data acquisition packets are found to have physical times falling within the same time window, the slot generation module extracts the underlying data payload of these packets and performs parsing. For speech recognition processing that transcribes the underlying audio stream into a natural language text sequence, and optical character recognition processing that locates and extracts text from an image pixel matrix, those skilled in the art can utilize existing acoustic models and machine vision pre-trained models. The underlying implementation principles are well-known technologies in this field and will not be elaborated upon here.

[0068] After parsing, the slot generation module calls the built-in clinical knowledge graph to map the parsed text features, image features, or numerical features to standardized clinical terms. For example, when the parsed speech text contains the semantic information of "increased wound exudation," and a patient's wound area image frame is detected in the visual data acquisition package within the same time window, the slot generation module merges the aforementioned speech semantic segment and the wound image together and maps them to the preset medical record field name "Wound Condition" as candidate content to be confirmed for that field.

[0069] S204, the slot generation module instantiates a medical record slot object containing a multimodal source reference pointer in memory, establishing the physical binding relationship between the structured fields and the original data.

[0070] Based on the above merge and mapping calculation results, the slot generation module instantiates specific structured medical record slots in the system's runtime memory. This structured medical record slot is defined as a data structure containing candidate values, timestamps, and source reference parameters, and its structure is expressed by the formula: ; in, This represents the generated structured medical record slot; This indicates the unique slot identifier assigned to this slot by the system; Indicates the global round identifier currently associated; Indicates the patient's identity; This indicates the names of the medical record fields obtained from the aforementioned mapping, specifically including chief complaint, present illness history, physical signs, wound condition, examination findings, medication records, medical advice and recommendations, and items to be followed up. This represents a set of multiple candidate values ​​extracted through parsing. This represents the set of reference parameters for multimodal sources; This represents the set of data acquisition timestamps corresponding to each candidate value; This represents the set of confidence scores used for subsequent calculations. This indicates the operational status parameters used to record the doctor's review actions; This indicates the version number of the current slot data record.

[0071] When instantiating the structured medical record slot, the slot generation module extracts the globally unique record identifiers of each heterogeneous data acquisition package within the same time window and stores them sequentially into the multimodal source reference parameter set. The pointers within this reference set point to specific audio segments of speech cycles, specific visual image frames, optical character recognition coordinate regions, or single vital sign interface numerical records at the underlying storage level. This object attribute-level physical binding mechanism ensures that every structured data item in the electronic medical record has a clear physical source, thereby establishing a complete data link that traces back from abstract medical record fields to the original multimodal scene evidence.

[0072] See attached document Figure 5 After generating structured medical record slots carrying source reference parameters, the system performs conflict detection on multiple candidate values ​​collected in the slot through weighted calculation and consistency comparison. This is specifically achieved through the following steps: S301, the confidence fusion and conflict detection module parses the multimodal source references in the structured medical record slots, extracts the calculation parameters of each independent data source, and performs weighted fusion to obtain the fusion confidence score.

[0073] The confidence fusion and conflict detection module reads the structured medical record slots generated in the preceding steps and extracts multiple candidate values ​​and their associated source reference parameters. For each underlying data source, the confidence fusion and conflict detection module extracts specific parameters for calculation. When the data source is speech text, the module extracts the word-level posterior probability output by the speech recognition model as the algorithm model's output score. When the data source is a visual image, the module obtains the predicted probability value output by the optical character recognition or object detection algorithm as the algorithm model's output score. For the specific implementation of obtaining the speech posterior probability or the visual model's predicted probability value, those skilled in the art can use the evaluation algorithms included in the deep neural network inference framework, which are well-known technologies in the field and will not be elaborated upon here.

[0074] Simultaneously, the confidence fusion and conflict detection module extracts reliability indices, temporal distance weighting parameters, and acquisition quality parameters from each data source. The reliability indices are preset based on the physical attributes and objectivity of the data source; for example, the reliability index for objectively measured vital signs data is set between 0.8 and 1.0, while the reliability index for speech and visual data is set between 0.4 and 0.7. The temporal distance weighting parameter is calculated based on the temporal distance decay between the physical time of acquisition and the center point of the time window for each data source. The confidence fusion and conflict detection module integrates the above multi-dimensional parameters and performs multi-source confidence weighted fusion calculation on the structured medical record slots. This calculation process is expressed by the formula: ; in, This represents the calculated slot fusion confidence score; This indicates the total number of candidate values ​​contained in the current structured medical record slot; Indicates the index number of the candidate value; This indicates the range from the 1st to the 2nd. Perform a cumulative summation operation on the corresponding values ​​of each candidate value; Indicates the first Reliability index of candidate values; Indicates the first The algorithm model outputs a score corresponding to each candidate value; Indicates the first The time distance weighting parameter for each candidate value; Indicates the first The acquisition quality parameters for each candidate value. After calculation, the confidence fusion and conflict detection module records the fusion confidence score of the slot in the corresponding structured medical record slot.

[0075] S302, the confidence fusion and conflict detection module compares the content consistency of different candidate values, and determines and outputs conflict slot records by combining the set numerical tolerance deviation and confidence threshold.

[0076] Since a single structured medical record slot aggregates candidate values ​​from multiple dimensions, the confidence fusion and conflict detection module needs to perform cross-validation on these candidate values. For numerical candidate values, the module extracts and calculates the absolute value of the difference between values ​​obtained from different sources. For example, the system compares the monitor readings extracted by optical character recognition with the underlying values ​​directly returned from the vital signs interface. For text-type candidate values, the module calls a natural language processing model to calculate the semantic similarity of different text fragments to check for logical contradictions between the spoken content from speech recognition and the visual features from image recognition.

[0077] The confidence fusion and conflict detection module presets a numerical conflict tolerance threshold and a fusion confidence threshold in memory. The numerical conflict tolerance threshold is set by those skilled in the art based on the reasonable error and fluctuation range of routine clinical physiological indicators, and is between 2% and 8% of the baseline value of the specific indicator. The fusion confidence threshold is set by those skilled in the art to be between 0.85 and 0.95, based on the mandatory limitations on the accuracy of medical record data in clinical systems. When the absolute value of the difference between numerical candidate values ​​exceeds the set numerical conflict tolerance threshold, or when the semantic similarity of text candidate values ​​indicates a cross-modal logical contradiction, the confidence fusion and conflict detection module determines that the difference between the current candidate values ​​is too large.

[0078] Simultaneously, when the slot fusion confidence score is calculated according to the aforementioned formula... When the data extraction result for that structured medical record slot falls below the set fusion confidence threshold, the system also determines it as a low-confidence candidate value. Upon detecting that any of the above conditions are met, the confidence fusion and conflict detection module marks the structured medical record slot as pending and outputs a conflict slot record with a pending conflict marker. This mechanism effectively intercepts unreliable data items, preventing them from being directly written to the final archive database, thus providing clear interception targets and triggering conditions for subsequent manual review and short-circuit confirmation logic.

[0079] After the confidence fusion and conflict detection module outputs conflict slot records with pending conflict markers, the system provides an interactive interface for operators and ensures the accuracy of the final medical record data through short-circuit confirmation logic. This is achieved through the following steps: S303, the doctor confirmation module captures voice commands, gesture interactions or key signals triggered by the operator in response to conflict slot records, and parses them to generate standardized confirmation action objects.

[0080] When the confidence fusion and conflict detection module generates conflict slot records, the AR prompt module renders a visual prompt for confirmation in a non-interference manner within the AR display module of the augmented reality smart glasses' waveguide. This prompt includes specific medical record field names and contradictory candidate values ​​extracted from different modalities. The doctor confirmation module is simultaneously activated, continuously listening for and capturing interaction signals triggered by the operator.

[0081] When an operator reviews via voice, the doctor confirmation module identifies and extracts preset confirmation keywords from the audio stream captured by the microphone array. These keywords include "confirm," "accept," "modify," and "re-capture." When an operator reviews via gesture, the doctor confirmation module analyzes the video stream from the first-person view camera module and identifies predefined specific gestures, such as a thumbs-up indicating acceptance and a horizontal wave indicating ignoring. When an operator reviews via button, the doctor confirmation module receives a hardware interrupt signal from the physical button on the augmented reality smart glasses. For the specific implementation of capturing and parsing voice commands and gesture signals, those skilled in the art can use mature speech recognition engines and gesture recognition model libraries; the implementation principles are well-known in the field and will not be elaborated upon here.

[0082] The doctor confirmation module will capture valid interaction signals, parse and encapsulate them into a unified confirmation action object. The data structure of this confirmation action object is represented by the following formula: ; in, This represents the confirmation action object generated by the doctor confirmation module; This indicates the slot identifier for the current operation; This indicates the parsed operation type, and its values ​​include adoption, modification, or re-collection. This indicates the final confirmed value entered by the doctor via voice or virtual keyboard when performing the modification operation; This indicates the physical moment when the confirmed action was captured.

[0083] S304, the doctor confirmation module executes short-circuit confirmation logic based on the confirmation action object, forcibly overwriting the machine-calculated value with the doctor's review result, and updating the status of the structured medical record slot.

[0084] After obtaining the standardized confirmation action object, the doctor confirmation module retrieves the unique slot identifier in memory. The corresponding structured medical record slot. The system then triggers short-circuit confirmation logic. Before triggering the short-circuit confirmation logic, the system verifies whether the patient identifier associated with the confirmation action object matches the currently active rounds session object, and whether the data writing window is open. If the verification passes, the short-circuit logic establishes the highest priority for doctor confirmation, directly bypassing and overwriting the confidence calculation results previously performed by the machine; if the verification fails, the short-circuit confirmation logic is not triggered and an alarm is generated.

[0085] The doctor confirmation module determines the type of operation in the confirmation action object. Perform the specific data update task. If the operation type is "adopt," the system selects the value specified by the doctor from the original candidate value set as the final result. If the operation type is "modify," the system will confirm the final confirmed value in the action object. As the final result, the doctor confirmation module writes this final result into the structured medical record slot, and simultaneously records the operation status parameters of this slot. The value has changed from "pending" to "confirmed".

[0086] After coverage is completed, the slot fusion confidence score and other candidate values ​​previously calculated by the confidence fusion and conflict detection modules are downgraded to audit references and are retained in the archived logs but no longer affect the formal medical record content. Meanwhile, the version number of the structured medical record slots... The system executes an increment operation to record this human intervention. This interactively triggered short-circuit confirmation and overwrite logic ensures that the structured data written into the final electronic medical record remains consistent with the doctor's professional judgment, achieving an effective closed loop between machine intelligence assistance and human expert decision-making.

[0087] See attached document Figure 6 During the continuous acquisition of structured medical record slots, the system calculates the consultation progress in real time by comparing with preset templates, and generates targeted supplementary tasks when key data omissions are found. This is achieved through the following steps: S401, the outpatient visit gap detection module reads the preset department standard template and the confirmed structured medical record slots, and performs set comparison operation to extract the current outpatient visit data gaps.

[0088] When doctors conduct rounds, the round gap detection module reads a pre-set departmental standard template for the current ward and the patient's diagnostic direction from the system's storage. The round gap detection module parses the departmental standard template, extracts the routine clinical features that must be obtained under the current consultation logic, and aggregates the field names of these features into a target slot set.

[0089] Simultaneously, the round-trip gap detection module monitors the in-memory data records within the current dedicated round-trip session object, retrieves structured medical record slots whose operation status parameters have changed to "confirmed," extracts the medical record field names corresponding to these confirmed slots, and aggregates them to form a confirmed slot set. After extraction, the round-trip gap detection module performs a set comparison operation in memory between the target slot set and the confirmed slot set. The system uses difference logic to filter out element records that exist in the target slot set but are missing in the confirmed slot set, thus obtaining the round-trip data gap set for the current round-trip task. This round-trip data gap set identifies structured medical record fields that have not yet been acquired or have not received valid confirmation in this session.

[0090] S402, the outpatient visit gap detection module calculates the data completeness parameter based on the gap in outpatient visit data, and generates follow-up questions or supplementary shooting tasks with action types and priorities by combining medical risk weights.

[0091] After obtaining the set of gaps in the outpatient data collection, the gap detection module calculates the ratio of the number of confirmed slots to the total number of target slots, thereby generating a data integrity parameter characterizing the current progress of patient collection. This data integrity parameter is continuously updated for the system backend to monitor the compliance of current medical records. For each specific missing field in the outpatient data gap set, the gap detection module instantiates the corresponding follow-up question or supplementary recording task in memory. The data structure of this task object is represented by the formula: ; in, This represents the generated task object; Indicates the task identifier assigned by the system; Indicates the action type set in the parsing; This indicates the generated text prompt field; This indicates the calculated task priority.

[0092] The gap detection module for outpatient visits determines the specific action type based on the underlying data modality upon which the missing field depends. When the missing clinical feature is primarily extracted through speech recognition (such as present medical history and medication experience), the module sets the action type as a follow-up questioning task. When the missing feature requires acquisition through a first-person perspective camera (such as the appearance of the affected area or laboratory test results), the module sets the action type as a re-capture task. The system extracts the specific name of the feature and assembles it to generate the content of the text prompt field, such as generating a natural language string like "Follow up on the current pain rating" or "Re-capture the drainage tube scale."

[0093] The round-call gap detection module calculates the priority value of a task based on the medical risk weights and corresponding missing durations configured in the departmental standard template. The medical risk weights are set between 0.1 and 1.0 by the backend management system according to the rigid requirements of clinical treatment pathways; a higher value indicates that the data is more critical to the diagnosis of the patient's condition. The missing duration is defined as the difference between the current physical time and the time the session started. The round-call gap detection module multiplies the medical risk weights and the missing durations to obtain the priority value and assign it to the task object's task priority. Once the task object is generated, it is pushed to the shared task scheduling queue by the system, providing basic data support for issuing accurate interactive prompts to operators in subsequent processes.

[0094] After generating follow-up questions or re-shooting tasks, the system intelligently schedules the timing of issuing prompts based on the operator's current operational status to avoid interfering with the normal diagnosis and treatment process. This is achieved through the following steps: S403, the doctor's status recognition module analyzes the environmental sound field features and multimodal pose data, determines and outputs a status identifier record representing the operator's current behavior.

[0095] The doctor's status recognition module continuously receives multi-dimensional environmental data collected by the system's underlying hardware. For the acoustic dimension, the module reads the environmental sound field features acquired by the microphone array and performs voice activity detection calculations. The system extracts the energy distribution and voiceprint direction of the acoustic signals to determine whether the operator is currently in a voice cycle communicating with a patient or verbally giving medical instructions. For the specific implementation of voice activity detection and sound source localization from environmental audio, those skilled in the art can use existing endpoint detection and beamforming algorithms, the principles of which are well-known in the field and will not be elaborated upon here.

[0096] Regarding the spatial posture dimension, the doctor's state recognition module analyzes video frames captured by the first-person view camera module and pose sequences output by the inertial sensor. The system extracts images of the operator's hand area from the video frames, identifies whether the operator is wearing sterile gloves, and detects whether there are any hand-holding actions involving medical instruments within the field of view. Combined with the specific head tilt angle reflected by the inertial sensor, the doctor's state recognition module determines whether the operator is performing routine clinical actions requiring high concentration, such as wound cleaning or dressing changes. For the specific process of extracting hand features from the video stream and performing target detection, those skilled in the art can utilize image classification models based on convolutional neural networks; the network structure and reasoning logic are well-known technologies in the field and will not be elaborated upon here.

[0097] Based on the above calculation results, the doctor status recognition module generates and outputs the current operator's status identifier record in memory. The specific values ​​of this status identifier record are divided into voice communication status, aseptic operation status, and idle status, and are synchronized to the main control computing bus in real time for downstream modules to access.

[0098] S404, the AR prompt module performs queuing and scheduling of follow-up questions or reshooting tasks based on the status identifier record, and extracts the text prompt fields that meet the issuance conditions for visual rendering output.

[0099] The AR prompting module receives follow-up questions or re-capture tasks submitted to the shared queue in the aforementioned steps, and simultaneously reads the status identifier records output in real time by the doctor's status recognition module. To avoid non-critical information interrupting the doctor's focused operation, the AR prompting module sets a criticality priority threshold in memory. This criticality priority threshold is set by those skilled in the art based on the intervention requirements of the clinical system for critical values, and its value ranges from 0.8 to 0.9. The AR prompting module executes the task issuance judgment logic based on the status identifier records and the task priority value of the follow-up questions or re-capture tasks. This judgment logic is expressed by the formula: ; in, This represents the system's output decision value. A value of 1 indicates that distribution is allowed, while a value of 0 indicates that distribution is postponed. This indicates the condition when the postcondition is true. This indicates the currently read status identifier record; Indicates the preset idle state indicator; This indicates a logical OR operation, which is triggered when either of the preceding or following conditions is met. This indicates the task priority value for follow-up questions or reshoots. This indicates the critical priority threshold set by the system. This indicates all other situations where the conditions specified above are not met.

[0100] When the system calculates a decision value of 0, meaning the operator is in a voice communication or aseptic operation state and the current task is not critical, the AR prompting module proactively keeps the follow-up question or re-capture task in the waiting queue, delaying its visual prompting timing. When the calculated decision value is 1, meaning the operator is idle, or the task priority value of the follow-up question or re-capture task is greater than or equal to the critical priority threshold, the AR prompting module extracts the corresponding text prompt field from the first-ranked follow-up question or re-capture task. The AR prompting module then sends this text prompt field to the waveguide AR display module on the augmented reality smart glasses, driving the underlying display driver chip to perform the corresponding visual rendering output. This state-aware scheduling mechanism ensures the effectiveness of clinical information distribution and achieves uninterrupted guidance of the operator's attention in busy rounds scenarios.

[0101] See attached document Figure 7 After structuring and filling gaps in the rounds data, the system, while ensuring the security of patient privacy data, introduces external evidence-based medicine knowledge through an intelligent routing mechanism to assist clinical decision-making. This is achieved through the following steps: S501, the edge-side desensitization module calls the NPU edge AI acceleration module to perform masking operations on the retained images and text, remove identity-sensitive entities, and generate desensitized query request data.

[0102] Before initiating any retrieval operation involving external data interaction, the edge-side de-identification module performs a cleaning process on the medical record text and multimodal stored images generated during the current session locally on the augmented reality smart glasses. The system calls the NPU edge AI acceleration module integrated into the hardware layer to load a pre-trained lightweight privacy-preserving model. For text-type data, the edge-side de-identification module runs a named entity recognition algorithm to scan the medical record text to extract character sequences containing information such as patient name, ID number, and ward / bed number. The edge-side de-identification module replaces the original content of these character sequences with preset placeholders or asterisks, completing the text-level de-identification operation.

[0103] For image-type data, the edge-side desensitization module runs object detection and semantic segmentation algorithms to locate pixel bounding boxes containing the patient's facial features and specific birthmarks or tattoos on the body surface within the image frame. The edge-side desensitization module performs visual masking operations such as Gaussian blur or pixel block replacement on the pixel matrix within these bounding boxes. For the specific computational process of performing named entity recognition and object detection based on neural networks, those skilled in the art can use mainstream edge-side deep learning inference frameworks; the algorithm implementation principles are well-known in the field and will not be elaborated here.

[0104] After the bimodal masking operation described above, all privacy information in the original data that could identify a specific patient was removed. The client-side desensitization module retains the desensitized key medical terms, structured medical record slot fields, and de-identified lesion features, and encapsulates this high-value clinical information to generate desensitized summary data and desensitized query request data for subsequent retrieval interactions.

[0105] S502, the de-identified RAG routing module receives de-identified query request data, evaluates the hit rate of the local offline knowledge base and the current network status, and executes the end-to-cloud adaptive routing strategy to initiate evidence-based medicine data retrieval.

[0106] After obtaining the de-identified query request data output by the edge-side de-identification module, the de-identified RAG routing module determines the specific execution node for the retrieval task. The de-identified RAG routing module extracts medical keywords from the query request and calculates vector similarity with the local offline knowledge base index stored in local physical memory. This yields the local offline knowledge base hit rate, which assesses the degree to which the local knowledge base satisfies the current query. Simultaneously, the system sends probe data packets through the underlying wireless communication module to obtain the current edge-cloud network latency parameters, serving as a specific indicator for evaluating the current network status.

[0107] The de-identified RAG routing module has preset hit rate and network latency thresholds in memory. The hit rate threshold is set by those skilled in the art to be between 0.70 and 0.85 based on the coverage breadth of the local knowledge base. The network latency threshold is set to be between 100ms and 200ms based on the timeliness requirements of clinical searches. The de-identified RAG routing module performs edge-cloud adaptive routing strategy calculations based on the above parameters, and its decision logic is expressed by the formula: ; in, This represents the routing decision node calculated and output by the system. Indicates a cloud-based knowledge base node; This indicates the condition when the postcondition is true. This represents the calculated hit rate of the local offline knowledge base; This indicates the hit rate threshold set by the system. This represents the logical AND operation, meaning that both the preceding and following conditions must be satisfied simultaneously. This indicates the current network latency parameter obtained; This indicates the network latency threshold set by the system. Indicates a local offline knowledge base node; This indicates other situations where the specified conditions are not met.

[0108] When the output routing decision node is a cloud-based knowledge base node, it indicates that local knowledge is insufficient to answer the current query and the network condition is good. The de-identified RAG routing module sends the de-identified query request data to the hospital's cloud-based inference server, initiating a large-model-driven cloud-based evidence-based medicine data retrieval. When the output routing decision node is a local offline knowledge base node, it indicates that the network environment is poor or local knowledge is sufficient to handle the current query. The de-identified RAG routing module downgrades the routing, directly retrieving the local offline knowledge base on the device side to initiate a local evidence-based medicine data retrieval. This adaptive routing strategy ensures that the device can continue to work in complex ward environments with poor wireless signal coverage, while also preventing the security risk of sensitive, un-de-identified data being leaked to the cloud.

[0109] After initiating an evidence-based medicine data retrieval and obtaining external support, the system deeply integrates the retrieval results into the medical record fields and completes the final data traceability and archiving, which is achieved through the following steps: S503, the de-identified RAG routing module parses the retrieved results, extracts medical advice citation information, and inserts it as evidence links into the corresponding structured medical record slots.

[0110] The de-identified RAG routing module receives retrieval response data streams from cloud-based large model nodes or local offline knowledge bases. The system parses this response data stream, extracting medical advice citation information from the structured return body. This information specifically includes the text of the treatment guidelines for the current condition, recommended medication interventions, and the names of the authoritative clinical guidelines referenced. The de-identified RAG routing module encapsulates the extracted parameters into an evidence link object, the data structure of which is represented by the formula: ; in, This represents the evidence link object generated by the system encapsulation; This indicates the slot identifier that triggered this evidence-based retrieval request; This represents the extracted medical advice text; This indicates the source information of the guidelines that support this evidence-based approach; This indicates the confidence level of the suggestion output by the external knowledge base or the local model.

[0111] The de-identified RAG routing module uses this slot identifier. The system precisely locates the corresponding structured medical record slot in system memory. Instead of using separate pop-ups or notifications to disrupt the operator's view, the system directly writes the encapsulated evidence link object as an extended attribute into the memory object of that structured medical record slot, thus backfilling the evidence-based basis. The system synchronously compares the medical recommendation text with the preliminary calculation results or candidate values ​​already existing in the current medical record slot. If there is a significant discrepancy in medical logic, the de-identified RAG routing module updates the status of the current structured medical record slot to "pending review" and provides the evidence-based recommendation as a new candidate version for doctors to access during their free time.

[0112] S504, the audit archiving module summarizes all related data objects generated in the current rounds session and merges them to generate a structured archive file with dimensional association attributes.

[0113] When the rounds session management module determines that the current patient's rounds have ended and the data writing window is closed, the audit archiving module takes over all operation records within the current dedicated rounds session object. The audit archiving module traverses the memory cache and extracts the structured medical record slots whose status parameters indicate they have been confirmed. The system simultaneously extracts gap collection records retained during the operation of the rounds gap detection module, manual review logs generated by the doctor confirmation module capturing interaction signals, and the evidence-based traceability links backed into the slots in the aforementioned steps.

[0114] The audit archiving module performs hierarchical nesting processing on the various objects collected above, according to the logical hierarchy between data elements. It sets the session identifier and operator identifier of the current round of visits as the root node of the data tree, and sets the structured medical record slots after confirmation as branch nodes. Based on this, the system binds the physical storage pointers of the multimodal data source, the manual review logs, and the evidence-based traceability links as leaf nodes of the corresponding branch nodes. The audit archiving module serializes the above-mentioned tree-structured memory objects and merges and packages them into a structured archive file for the current round of visits. For the specific implementation of serializing memory-level objects into standard data exchange files, those skilled in the art can use unified text encoding protocols and format specifications; the data assembly and parsing principles are well-known technologies in the field and will not be elaborated here.

[0115] The structured archive files ultimately output by the system possess complete dimensional correlation attributes. These attributes ensure that every diagnostic conclusion recorded in the electronic medical record system can be precisely traced back to a specific attending physician and the physical time of ward rounds at the top level. At the bottom level, these attributes allow the system to trace and reproduce the original medical order recordings extracted by the microphone array, the image slices of the affected area captured by the camera, and the original readings output by the medical instrument interface at any time. This end-to-end traceability mechanism effectively meets the audit requirements for preventing tampering of clinical data and determining liability for medical malpractice.

[0116] To further aid in understanding the proposed solution, a specific application example is provided here, tailored to a real-world clinical scenario. In the respiratory ward of a tertiary hospital, the attending physician wears augmented reality smart glasses equipped with the proposed solution to conduct routine mobile rounds on the patient in bed 3.

[0117] The doctor enters the ward and focuses on bed 3. The first-person view camera module captures the current bed area. The patient bed identification module extracts the QR code at the head of the bed and the spatial positioning features of the ward, performs cross-validation, and generates identification status data. The rounds session management module instantiates a dedicated rounds session object for the patient in bed 3 in the system's memory. Simultaneously, the vital signs device dynamic binding module scans the broadcast packets from the bedside monitor to extract signal strength parameters, verifies them against the logbook, establishes a connection, and outputs binding status data. Based on the above status data, the system writes the data to the window and sets it to the open state.

[0118] The doctor asks the patient via voice, "Are you still feeling chest tightness today?" The multimodal acquisition module simultaneously acquires the voice dialogue, images from the medical record folder and examination sheets captured by the first-person camera module, and receives the blood oxygen saturation readings from the monitor via the vital signs interface. The slot generation module aggregates heterogeneous data packets within the same time window according to a set time window segmentation standard, calls the semantic model to map these data to specific medical record field names, and generates structured medical record slots carrying multimodal source reference parameters.

[0119] The confidence fusion and conflict detection module performs conflict detection on the acquired feature data. If there are modal differences between the candidate content of speech recognition and image recognition, the system reads the corresponding reliability indicators and other calculation parameters to obtain the slot fusion confidence score. When the difference exceeds the limit or the score is too low, the system outputs a conflict slot record with an pending conflict marker. After seeing the prompt in the doctor's field of vision, the doctor verbally commands "accept the instrument value". The doctor confirmation module captures this command, parses the confirmation action object, and then triggers the short-circuit confirmation logic to overwrite the machine-calculated value, marking the corresponding structured medical record slot as confirmed.

[0120] The rounds gap detection module reads the respiratory medicine department's standardized template and performs a difference comparison with the confirmed slots to generate rounds data gaps indicating missing drainage fluid scale features. It calculates the current data completeness parameter and generates follow-up questions or re-capture tasks based on medical risk weights. At this time, the doctor's status recognition module identifies that the doctor's hands are in a sterile operating state and generates a status identifier record. The AR prompt module, based on its judgment logic, temporarily suspends processing this task and performs queuing scheduling until the doctor's actions are completed, at which point it extracts the text prompt field and renders it.

[0121] After leaving the ward, the edge-side de-identification module invokes the NPU edge AI acceleration module to perform masking operations on the retained medical records and images to generate de-identified query request data. Subsequently, the de-identification RAG routing module assesses that the local offline knowledge base hit rate is lower than the system's set hit rate threshold, and the current network latency parameters are relatively good. Based on the edge-cloud adaptive routing strategy, it determines to initiate a retrieval to the cloud knowledge base node. The medication intervention measures returned by the system are parsed and used as evidence links to populate the slots. Finally, the audit archiving module summarizes and nests all operation records to generate a final structured archive file with complete dimensional correlation attributes.

[0122] To verify the robustness and intelligent scheduling efficiency of the system modules in complex clinical environments, the following experimental comparisons were conducted using the formulas provided in the specific implementation of this application.

[0123] According to the calculation formula in the specific implementation method: ; Three modes are set ( Extract candidate values ​​for a specific vital sign item: speech ( ), visual ( ), vital signs instruments ( Based on the aforementioned value range, the reliability index is set as follows: (voice), (Visual) (Instrument); Assuming the data acquisition occurs within the same reference time window, the time distance weighting parameter... All values ​​are 1, and the acquisition quality parameters are set to 1. The algorithm model output score of the vital signs instrument is constant. The system's fusion confidence threshold is set to 0.85.

[0124] See attached document Figure 8 The horizontal axis represents the output score of the speech algorithm model, ranging from 0.5 to 1; the vertical axis represents the slot fusion confidence score, also ranging from 0.5 to 1. The attached figure contains three reference lines: a dashed line with a triangle representing the single-modal prediction score, a solid line with a circle representing the multi-source weighted fusion score (in this application), and a horizontal dotted line at 0.85 on the Y-axis representing the fusion confidence threshold.

[0125] As shown in the figure, both the single-modal prediction score and the multi-source weighted fusion score exhibit a linear upward trend with the increase of the speech algorithm model output score. The single-modal prediction score shows a steeper upward slope; when the speech score on the horizontal axis drops below 0.85, its score is correspondingly below the fusion confidence threshold. The multi-source weighted fusion score curve of this application has a smaller slope and remains at a higher overall level; even when the horizontal axis drops to around 0.65, the curve still maintains above the fusion confidence threshold of 0.85.

[0126] The aforementioned discrepancies arise because environmental interference in the ward environment can cause a decrease in the output score of single-modality speech or vision algorithm models, and single-modality models are prone to false alarms due to brief disturbances. This application employs a multi-source confidence-weighted fusion mechanism, utilizing highly reliable and high-quality objective instrument data in the weighted calculation. This reduces the interference of low-quality data sources on the final result, maintaining a high level of confidence score for the overall calculated slot fusion. This mechanism reduces the system's false alarm rate and decreases the manual review workload of the doctor confirmation module.

[0127] According to the calculation formula for priority definition in the specific implementation method: The system selects three feature items from the department's standardized template to generate gaps in the rounds data: Task A (body temperature and blood pressure, high medical risk weight 0.25), Task B (routine appearance imaging, medium medical risk weight 0.12), and Task C (subjective dietary experience, low medical risk weight 0.05). The system then uses a decision-making formula based on these criteria. ; Set the criticality priority threshold for system intervention. It is 0.85.

[0128] See attached document Figure 9 The horizontal axis represents the missing duration measurement unit, with a value range of 1 to 10; the vertical axis represents the task priority value, with a value range of 0 to 3. The attached figure contains four reference lines: a solid line with a triangle marking representing task A (high medical risk weight), a dashed line with a circle marking representing task B (medium medical risk weight), a dotted line with a square marking representing task C (low medical risk weight), and a horizontal dotted line at 0.85 on the Y-axis representing the critical priority threshold.

[0129] As shown in the graph, the priority values ​​of all three tasks exhibit a linear upward trend with the increase in missing duration measurement units, but the slopes of the rising lines differ. Task A has the steepest slope, exceeding the 0.85 threshold when the missing duration reaches approximately 3.4 measurement units. Task B's upward trend is moderate, exceeding the threshold between the 7th and 8th measurement units. Task C has the smallest slope; within the observation period of 10 measurement units, its value remains in the lower range and does not reach the threshold.

[0130] The reason for this trend is that, according to the calculation formula, the task priority value is derived by multiplying the medical risk weight by the missing duration. Therefore, the magnitude of the medical risk weight directly determines the slope at which the priority value increases over time. A high medical risk weight causes the priority value of task A to increase faster, thus meeting the condition of reaching the critical priority threshold in the decision-making formula earlier, triggering the system to issue and render the output. The priority value of task C, with a low medical risk weight, increases more slowly, and the system will determine that it should be postponed and queued, pushing the notification when the operator is detected to be idle. This mechanism ensures the timeliness of the notification in the critical care screening process while reducing interference with routine clinical operations.

[0131] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0132] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0133] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing multimodal mobile medical data from AR smart glasses, characterized in that, Includes the following steps: The system identifies the patient's identity and generates identity recognition status data; it establishes a binding connection with the external detection equipment and generates binding status data; and it adjusts the data writing window to be open or suspended based on the identity recognition status data and the binding status data. While the data writing window is open, the system synchronously acquires dialogue voice, images of the affected area, and instrument values. It then performs aggregation processing on the heterogeneous data according to the time window segmentation standard to generate structured medical record slots carrying source reference parameters. Perform multi-source confidence weighted fusion calculation on multiple candidate values ​​in the structured medical record slots and output conflict slot records. Capture the interaction signals triggered by the conflict slot records and execute short-circuit confirmation logic to overwrite the machine-calculated values. The system reads the department's standard template and performs a set comparison operation with the acquired structured medical record slots to generate follow-up questions or reshoot tasks. It analyzes the environmental sound field characteristics and pose sequence to determine the operator's status, and performs queuing scheduling for the follow-up questions or reshoot tasks to extract text prompt fields and send them for rendering output. The edge AI acceleration module is invoked to perform masking operations to generate de-identified query request data, initiate evidence-based medicine retrieval, insert medical advice citation information into the corresponding structured medical record slots, and summarize to generate structured archive files.

2. The AR smart glasses multimodal mobile medical data processing method according to claim 1, characterized in that, Adjusting the data writing window to be open or suspended based on the identity recognition status data and the binding status data includes: Visual and spatial positioning features are extracted to identify the patient's identity and generate the identity status data. The low-power Bluetooth broadcast packets sent by the bedside peripheral device are scanned and combined with the patient's bed identification to verify and establish a binding connection relationship to generate the binding status data. When the identity matching score is greater than or equal to the matching score threshold and the device binding status parameter is in the bound state, the data writing window status parameter is changed to the open state. If the duration of no effective doctor-patient interaction is detected exceeds the silent duration threshold, the data writing window state parameter is changed to a suspended state, and the corresponding subsequent data is transferred to the isolation buffer for temporary storage.

3. The AR smart glasses multimodal mobile medical data processing method according to claim 1, characterized in that, The generation of structured medical record slots carrying source reference parameters includes: Simultaneously acquire the dialogue voice, affected area image and instrument values, and bind them with physical time and spatial positioning parameters to generate a heterogeneous data acquisition package; Extract heterogeneous data collection packets that fall within the same time window interval, perform parsing operations, and map them to medical record field names; Instantiate a structured medical record slot containing candidate values, timestamps, and source reference parameters, and store the globally unique record identifier within the same time window into the multimodal source reference parameter set.

4. The AR smart glasses multimodal mobile medical data processing method according to claim 1, characterized in that, The process of performing multi-source confidence weighted fusion calculation and outputting conflict slot records, and capturing interaction signals triggered by the conflict slot records to execute short-circuit confirmation logic, includes: The reliability index of independent data sources, the output score of the algorithm model, and the time distance weight parameter are extracted to perform multi-source confidence weighted fusion calculation to obtain the slot fusion confidence score; When the absolute value of the difference between multiple candidate values ​​is greater than the numerical conflict tolerance threshold, there is a cross-modal logic contradiction, or the slot fusion confidence score is lower than the fusion confidence threshold, the conflict slot record with the pending conflict mark is output. Based on the captured interaction signal parsing, a confirmation action is generated, and short-circuit confirmation logic is executed according to the confirmation action to force the verification result to overwrite the machine-calculated value.

5. The AR smart glasses multimodal mobile medical data processing method according to claim 1, characterized in that, The task of generating follow-up questions or reshoots includes: Extract routine clinical features from the department's standardized template and perform set comparison operations with the confirmed structured medical record slots to obtain the set of gaps in the rounds data; For the missing fields in the aforementioned outpatient data gap set, the action type is determined to be either a follow-up questioning task or a reshooting task. A priority value is calculated by combining the medical risk weight and the duration of the missing data, and this value is used to generate the follow-up questioning or reshooting task.

6. The AR smart glasses multimodal mobile medical data processing method according to claim 5, characterized in that, The process of determining the operator's status and performing queue scheduling, extracting text prompt fields, and sending them for rendering output for the follow-up questioning or reshooting tasks includes: The environmental sound field characteristics and the pose sequence of the inertial sensor are analyzed, and the status identifier records are output, which are divided into voice communication state, aseptic operation state, or idle state. When the current state is either voice communication or aseptic operation and the priority value is less than the critical priority threshold, the follow-up questioning or reshooting task will be kept in the waiting queue to postpone the visual prompting time. When the current state is idle or the priority value is greater than or equal to the critical priority threshold, the text prompt field of the first-ranked follow-up question or reshoot task is extracted and sent to the optical waveguide AR display module for visual rendering output.

7. The AR smart glasses multimodal mobile medical data processing method according to claim 1, characterized in that, The initiation of evidence-based medicine search includes: The NPU edge AI acceleration module is invoked to locate the area containing identity privacy information, perform a masking operation, and generate the de-identified query request data; Extract medical keywords from the de-identified query request data and perform vector similarity calculation with the local offline knowledge base index to obtain the local offline knowledge base hit rate; When the hit rate of the local offline knowledge base is less than the hit rate threshold and the network latency parameter is less than the network latency threshold, a cloud-based evidence-based medicine data retrieval is initiated; otherwise, a local evidence-based medicine data retrieval is initiated.

8. A multimodal mobile medical data processing system for AR smart glasses, characterized in that, The method for processing multimodal mobile medical data for AR smart glasses as described in any one of claims 1-7 includes: The patient bed identification module, the vital signs equipment dynamic binding module, and the round-trip session management module are used to identify patient identity identifiers to generate identity identification status data, establish binding connection relationships with peripheral detection equipment to generate binding status data, and adjust the data writing window to be open or suspended based on the identity identification status data and the binding status data. The multimodal acquisition module and slot generation module are used to simultaneously acquire dialogue voice, affected area images and instrument values ​​when the system is enabled. They also perform aggregation processing on heterogeneous data according to the time window segmentation standard to generate structured medical record slots carrying source reference parameters. The confidence fusion and conflict detection module and the doctor confirmation module are used to perform multi-source confidence weighted fusion calculation on candidate values ​​in the structured medical record slots and output conflict slot records, capture interaction signals and execute short-circuit confirmation logic to overwrite machine-calculated values; The outpatient gap detection module, doctor status recognition module, and AR prompt module are used to read the department standard template and the acquired structured medical record slots to perform set comparison operations to generate follow-up questions or reshoot tasks, analyze the environmental sound field features and pose sequence to determine the operator's status, and perform queuing scheduling for the follow-up questions or reshoot tasks to extract text prompt fields and send them for rendering output. The edge-side de-identification module, de-identification RAG routing module, and audit archiving module are used to call the edge AI acceleration module to perform masking operations to generate de-identified query request data, initiate evidence-based medicine retrieval, insert medical advice citation information into the corresponding structured medical record slots, and summarize to generate structured archive files.

9. The AR smart glasses multimodal mobile clinic data processing system according to claim 8, characterized in that, It also includes a physical hardware architecture built on AR smart glasses, which includes a main control SoC, an NPU edge AI acceleration module, an input link, and an output link; The input link includes a first-view camera module, a microphone array, an inertial sensor, a wear detection unit, and a vital signs interface; The output link includes an optical waveguide AR display module and a bone conduction speaker; The multimodal acquisition module is used to activate the input link to concurrently capture environmental audio, first-view images, and peripheral vital sign interface data. Based on a globally unified hardware clock system, it generates standardized data acquisition packages by binding physical time and spatial positioning parameters to heterogeneous data.

10. The AR smart glasses multimodal mobile clinic data processing system according to claim 8, characterized in that, The specific logic for the audit archiving module to generate structured archive files is as follows: Perform hierarchical nesting processing according to logical hierarchy, set the session identifier and operator identifier of the current round of visits as the root node of the data tree, and set the confirmed structured medical record slot as the branch node; The physical storage pointers of multimodal data sources, manual review logs, and evidence-based traceability links are bound to the leaf nodes of the corresponding branch nodes. The memory objects that present a tree structure are serialized, merged, and packaged into a structured archive file with dimensional association attributes.

Images