AI glasses audio-visual multi-language fusion inquiry guidance method and system

Abstract

The application relates to the technical field of AI glasses assisted diagnosis, and discloses an AI glasses audio-visual multi-language fusion diagnosis guidance method and system, which comprises the following steps: collecting images and audio signals to generate structured and multi-language semantic data; extracting medical sensitive fields in the multi-language semantic data to generate protection entities; aggregating data satisfying dynamic time window constraints into multi-modal event clusters; inputting a double-layer evidence mutual verification matrix to deduce clinical elements and conflict control signals; using high-credibility elements to obtain evidence-based context, combining the conflict control signals to generate operation guidance actions; and presenting and collecting feedback update state values on the glasses side. Through multi-modal aggregation and double-layer mutual verification deduction, the application realizes cross verification of visual and auditory clues and multi-language semantics, reduces the risk of model misinterpretation by using a sensitive field isolation confirmation mechanism, directly outputs operation instructions in combination with evidence-based knowledge, and realizes terminal closed loop, thereby improving the accuracy and safety of clinical diagnosis.

Description

Technical Field

[0001] This application relates to the field of AI glasses-assisted medical consultation technology, specifically to AI glasses audiovisual multilingual fusion medical consultation guidance methods and systems. Background Technology

[0002] With the application of artificial intelligence technology in the medical field, smart devices are gradually participating in clinical consultations and ward rounds, aiming to assist doctors in improving work efficiency. Existing assisted consultation systems mostly use speech-to-text transcription as the primary input information, but this approach has limitations. In actual bedside consultations, visual cues such as the patient's unspoken wound condition, abnormal posture, and numerical alarms from bedside devices are easily missed by the system. Furthermore, existing image or audiovisual recognition functions often operate independently, lacking a temporal alignment mechanism between visual representations, auditory features, and verbal statements. Due to the lack of aggregation and cross-validation capabilities for multimodal data under a unified temporal benchmark, the system struggles to detect factual conflicts between different modalities, resulting in incomplete and inaccurate extraction of clinical elements.

[0003] When dealing with patients speaking foreign languages ​​or dialects, conventional systems rely on general language models for translation. However, in complex medical contexts, the models are prone to semantic deviations or arbitrary rewriting when translating core medically sensitive information such as drug names, dosages, and symptom durations. Because current technologies lack dedicated isolation and mandatory verification processes for these sensitive fields, unverified erroneous translations can directly enter the underlying inference network and participate in fact-finding, potentially leading to biased medical judgments and increasing security risks in multilingual consultation scenarios.

[0004] Existing diagnostic aids are limited to outputting text-based diagnostic suggestions or case drafts, failing to translate clinical deductions into actionable instructions compatible with wearable devices. Doctors still need to manually analyze fragmented information and translate it into subsequent diagnostic and treatment actions during consultations, as the system cannot directly provide real-time prompts for follow-up questions, observations, or examinations within the doctor's field of vision. Furthermore, the lack of a closed-loop feedback mechanism for terminal interaction and status confirmation increases the number of steps doctors need to take to integrate multi-source information at the bedside, hindering its ability to truly provide real-time auxiliary guidance. Summary of the Invention

[0005] This application provides an AI glasses audio-visual multilingual fusion consultation guidance method and system, which solves the problems of existing auxiliary consultation systems relying on single text and missing visual cues, lacking cross-modal mutual verification, easily mistranslating sensitive medical information in cross-language translation and lacking isolation confirmation, and being unable to directly output executable operation guidance and form closed-loop feedback.

[0006] To achieve the above objectives, this application provides the following technical solution: The first aspect of this application provides a method for AI glasses to provide multilingual audiovisual fusion diagnostic guidance, including the following steps: First-person view images and environmental audio signals are acquired to generate structured data of visual events, structured data of auditory events, and multilingual semantic data; Extract medically sensitive fields from the multilingual semantic data to generate a medically sensitive field protection entity; The structured data of visual events, structured data of auditory events, multilingual semantic data, and the entities protecting medical sensitive fields that satisfy the dynamic time window constraints are aggregated to generate a multimodal event cluster. The multimodal event clusters are input into a two-layer evidence verification matrix to deduce clinical elements, and clinical elements or conflict control signals with high confidence acceptance labels are output. The clinical elements with high confidence acceptance markers are used to obtain an evidence-based context data set, and the conflict control signal and the evidence-based context data set are combined to generate operational guidance actions. The operation guidance actions are sent to the AI ​​glasses for display, feedback interaction signals are collected to generate a confirmation record, and the confirmation status value of the clinical element is updated based on the confirmation record.

[0007] Preferably, the generation of structured visual event data includes: Target detection and semantic segmentation are performed on the first viewpoint image. The boundary coordinate matrix of the target object is extracted to generate the region of interest reference parameter. The probability value of the target object's category is output as the confidence parameter. Call the morphological feature evaluation operator to map and output the severity level parameter; The target object, the region of interest reference parameter, the confidence parameter, and the severity level parameter are combined to generate structured data of the visual event. When the confidence parameter is lower than the safety tolerance threshold, the position of the confirmation flag is set to true.

[0008] Preferably, the generation of structured auditory event data and multilingual semantic data includes: The environmental audio signal is truncated and segmented to obtain effective audio segments. The effective audio segments are converted into an acoustic feature matrix, input into an audio classification network to identify non-morpheme sound events, and combined with the extracted timestamp parameters to generate auditory event structured data. The effective audio segment is input into a language classification network to determine the source language identifier, a language model is loaded to decode and generate the original text, a cross-language semantic mapping model is called to convert the original text into translated text, and the source language identifier, the original text and the translated text are structurally bound to generate multilingual semantic data.

[0009] Preferably, extracting medically sensitive fields from the multilingual semantic data to generate a medically sensitive field protection entity includes: The medical-specific dictionary and rule base of the language are called to perform sequence labeling calculation on the original text in the multilingual semantic data to extract medical sensitive fields, and encapsulate and generate a medical sensitive field protection entity containing a confirmation status value, and initialize the confirmation status value to an unconfirmed state; Check the confirmation status value of the medical sensitive field protection entity. For medical sensitive field protection entities in the unconfirmed state, block specific word content from participating in semantic understanding and fact inference calculations, and trigger an explicit interactive task to confirm the specific word content.

[0010] Preferably, the step of aggregating the structured visual event data, structured auditory event data, multilingual semantic data, and the medically sensitive field protection entity that satisfy dynamic time window constraints to generate a multimodal event cluster includes: Extract the temporal feature parameters of the visual event structured data, auditory event structured data, and multilingual semantic data, map them to an absolute time axis, and assign them a reference physical time coordinate; Identify the environmental audio signals to define the consultation cycle and generate a session identifier, and define the consultation rounds and generate round identifiers; The dynamic time window constraint is formed by setting forward time window thresholds and backward time window thresholds using the reference physical time coordinates. Data that fall within the dynamic time window constraint and meet the conditions of consistent round identifiers, time span not exceeding the maximum time span threshold, and consistent patient context feature signature parameters are associated and mapped to generate multimodal event clusters.

[0011] Preferably, the multimodal event clusters are input into a two-layer evidence verification matrix to deduce clinical elements, and clinical elements or conflict control signals with high confidence acceptance markers are output, including: The initialization of the pre-mutual verification matrix and the confirmation mutual verification matrix serve as the two-layer evidence mutual verification matrix; The medical logic reasoning network is invoked to extract clinical elements from the multimodal event cluster, calculate the support status of different modal channels for the clinical elements, and fill the basic evidence layer into the pre-mutual verification matrix. Assess the time compactness and state security corresponding to the support state, calculate the spatiotemporal consistency score, and fill the constraint verification layer of the pre-mutual verification matrix together with the cross-language translation consistency flag and the confirmation state value of the medical sensitive field protection entity. Based on the support status, multidimensional confidence calculation is performed on the clinical elements. For clinical elements whose confidence reaches the comprehensive confidence threshold, a high confidence acceptance flag is output, and for clinical elements containing conflicts, a conflict control signal is output.

[0012] Preferably, the clinical elements with high confidence markers are used to obtain an evidence-based contextual data set, and operational guidance actions are generated by combining the conflict control signal and the evidence-based contextual data set, including: The clinical elements with high confidence acceptance tags are subjected to medical concept normalization processing and mapped to standard medical concept codes; Extract the current patient's historical confirmation status records as a set of query constraints, and combine them with the standard medical concept encoding and depth limit threshold to encapsulate and generate a query request entity; The query request entity is used to initiate a call to the built-in evidence-based knowledge graph. Based on the standard medical concept encoding and depth limit threshold in the query request entity, graph traversal calculation is performed. Logical extension path pruning and filtering are performed according to the query constraint set to obtain the knowledge graph path entity containing associated target concept nodes and recommendation level weights as the evidence-based context data set. The priority parameters and action type enumeration values ​​are calculated based on the received conflict control signal and the evidence-based context data set, combined with a preset operation template. Fill the placeholders in the operation template with missing evidence descriptions or related target concept nodes to generate a structured operation guidance action that includes text prompt fields, action type enumeration values, and priority parameters.

[0013] Preferably, the operation guidance action is sent to the AI ​​glasses for display, feedback interaction signals are collected to generate a confirmation record, and the confirmation status value of the clinical element is updated based on the confirmation record, including: The priority parameters in the operation guidance actions are extracted, sorted in descending order, and serialized to output an ordered instruction queue. The text prompt field of the operation guidance action ranked first in the ordered instruction queue is sent to the AI ​​glasses for rendering and overlay. The system collects feedback signals from operators, classifies and calculates their execution intentions, and generates a confirmation record containing the feedback operation enumeration value and the new correction value parameter. Extract the traceability index parameters from the confirmation record to retrieve and locate the multimodal event cluster and its corresponding clinical element, and use the feedback operation enumeration value to update the confirmation status value of the clinical element in the constraint verification layer.

[0014] The second aspect of this application provides an AI glasses audiovisual multilingual fusion diagnostic guidance system, applied to the AI ​​glasses audiovisual multilingual fusion diagnostic guidance method described in the first aspect above, including: AI glasses are used to collect first-person view images and environmental audio signals; The first-view visual understanding module is used to generate structured visual event data based on the first-view image. The auditory understanding module is used to generate structured data of auditory events based on the environmental audio signals; A language recognition and medical translation module is used to generate multilingual semantic data based on the environmental audio signals; The medical terminology protection module is used to extract medically sensitive fields from the multilingual semantic data and generate a medically sensitive field protection entity. The cross-modal semantic alignment module is used to aggregate the structured data of visual events, structured data of auditory events, multilingual semantic data, and the medical sensitive field protection entities that meet the dynamic time window constraints to generate a multimodal event cluster. The evidence verification matrix module is used to input the multimodal event cluster into the two-layer evidence verification matrix to deduce clinical elements and output clinical elements or conflict control signals with high confidence acceptance marks. An evidence-based medicine intelligent agent adapter is used to acquire a set of evidence-based context data using clinical elements with high confidence acceptance tags; The doctor's operation guidance generation module is used to generate operation guidance actions by combining the conflict control signal and the evidence-based context data set; The glasses-side limited display module is used to send the operation guidance actions to the AI ​​glasses for display. The bilingual confirmation module is used to collect feedback interaction signals to generate confirmation records, which are then sent back to the evidence verification matrix module to update the confirmation status value of the clinical elements based on the confirmation records.

[0015] Preferably, the glasses-side limited display module is specifically used to parse the priority parameters of the operation guidance action, perform serialized queuing and scheduling, extract the text prompt field of the first-ranked operation guidance action and send it to the near-eye display component of the AI ​​glasses for visual rendering, and at the same time push the associated secondary evidence details into the display queue of the auxiliary terminal device; The bilingual confirmation module is specifically used to capture the voice, button or gesture feedback interaction signals of the operator in response to the operation guidance actions through the AI ​​glasses terminal, parse the feedback interaction signals into confirmation records of acceptance, modification, ignore or inapplicability, and write the confirmation records in reverse into the evidence mutual verification matrix module to cover the confirmation status value of the corresponding clinical elements.

[0016] This application provides a method and system for AI glasses audiovisual multilingual fusion diagnostic guidance, which has the following beneficial effects: 1. This application generates multimodal structured data by collecting first-person perspective images and environmental audio signals, aggregates multimodal event clusters using dynamic time window constraints, and inputs these multimodal event clusters into a two-layer evidence mutual verification matrix to deduce clinical elements and conflict control signals. Combined with the evidence mutual verification deduction mechanism, it solves the problem that traditional auxiliary diagnosis mainly relies on single text and misses visual abnormalities such as wounds, body posture, and equipment values. It realizes the alignment and cross-verification of visual cues, auditory cues, and multilingual semantics in the same judgment link, establishes fact mutual verification relationships between different modal channels, and improves the accuracy and comprehensiveness of clinical element acquisition.

[0017] 2. This application generates a medically sensitive field protection entity by extracting medically sensitive fields such as drug names, dosages, and symptom durations from multilingual semantic data. When the protection entity is in an unconfirmed state, specific word content is blocked from participating in subsequent semantic understanding and fact-inference calculations, and an interactive task for confirming specific word content is triggered. This introduces an isolation and confirmation process for medically sensitive fields, overcoming the defect that core medical information is easily mistranslated or freely rewritten by language models in cross-language or dialect communication. It also blocks biased translation results from entering the clinical reasoning network and reduces the risk of medical judgment in multilingual consultation scenarios.

[0018] 3. This application utilizes clinical elements with high-confidence acceptance tags to call knowledge graphs to obtain a set of evidence-based context data, combines conflict control signals to generate structured operation guidance actions, sends the operation guidance actions to the AI ​​glasses for presentation, and collects feedback interaction signals to update the confirmation status value. Combined with the deduction results of clinical elements, it generates specific operation instructions, which changes the problem of traditional systems lacking executable terminal guidance actions. It can directly output real-time observation, questioning and examination action prompts within the doctor's field of vision. Through terminal presentation and interactive confirmation closed loop, it reduces the number of operation steps for doctors to integrate multi-source information at the bedside. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0022] Figure 1 This is an architectural block diagram of the AI ​​glasses audiovisual multilingual fusion consultation guidance system of this application; Figure 2 This is a flowchart illustrating the steps of the AI ​​glasses audio-visual multilingual fusion consultation guidance method of this application; Figure 3 This is a flowchart of the multimodal audiovisual data acquisition and feature analysis process in this application; Figure 4 This is a flowchart illustrating the extraction and data isolation protection process for medically sensitive fields in this application. Figure 5 This is a flowchart of the cross-modal multidimensional coordinate calibration and event cluster aggregation process in this application; Figure 6 This is a flowchart illustrating the instantiation of the two-layer evidence mutual verification matrix and the state arbitration process in this application. Figure 7 A flowchart was generated for the evidence-based knowledge graph query and operation guidance actions in this application. Figure 8 This is a flowchart of the closed-loop process for end-side amplitude limiting display scheduling and interaction status confirmation in this application; Figure 9 This is a distribution diagram of the credibility score under the cross-modal dynamic time window constraint of this application; Figure 10 This is a comparison chart of the confidence levels before and after the closure of the evidence mutual verification matrix in this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0025] For ease of description, spatial relative terms may be used in the text to describe the relative position or movement of one element or feature relative to another element or feature, as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "below," "above," "front," "back," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure. For example, if the device in the figure undergoes a positional flip, orientation change, or change of motion, these directional indications will change accordingly. For instance, an element described as "below other elements or features" or "below other elements or features" will subsequently be oriented "above other elements or features" or "above other elements or features." Therefore, the example term "below" can include both upper and lower orientations. The device may be otherwise oriented (rotated 90 degrees or in other directions), and the spatial relative descriptors used in the text will be interpreted accordingly.

[0026] See attached document Figure 1 This application provides an AI glasses audio-visual multilingual fusion consultation guidance system, including an AI glasses terminal, a first-person visual understanding module, an auditory understanding module, a language recognition and medical translation module, a medical terminology protection module, a cross-modal semantic alignment module, an evidence mutual verification matrix module, an evidence-based medicine intelligent agent adapter, a doctor operation guidance generation module, a glasses terminal limited display module, and a bilingual confirmation module.

[0027] The AI ​​glasses are used in clinical consultations, ward rounds, or outpatient visits to collect first-person view images and environmental audio signals from the scene through a built-in first-person view camera and microphone array. The text prompts are rendered and output through the near-eye display component, and the collected multimodal data is transmitted to the processing computing node through the communication link.

[0028] The first-person perspective visual understanding module receives first-person perspective images and uses visual recognition algorithms to detect and classify high-value medical targets such as wounds, examination reports, and bedside instrument readings in the first-person perspective images. It extracts visual features and outputs structured visual event data containing region of interest reference parameters, severity, and confidence parameters.

[0029] The auditory understanding module is used to parse non-linguistic sound events in environmental audio signals, classify environmental audio signals into non-linguistic sound events such as coughing, wheezing, or instrument alarm sounds, and output structured data of auditory events with timestamp parameters.

[0030] The language recognition and medical translation module is used to extract morphemes from environmental audio signals, perform language determination and speech-to-text transcription calculations, call a cross-language semantic mapping model to convert the patient's original speech into target language text, and output multilingual semantic data including the original text, translated text, source language identifier, timestamp parameters, as well as a set of medical terms, a set of negative words, symptom time expressions, confidence level and translation status.

[0031] The medical terminology protection module is used to extract sensitive medical fields such as drug names, dosage parameters, allergy history records, and negative semantic expressions from the original text based on a preset medical dictionary. It generates independent protection identifiers for the sensitive medical fields and assigns them an initial unconfirmed state. In subsequent data flow nodes, a data blocking and isolation mechanism for the unconfirmed state is established.

[0032] The cross-modal semantic alignment module is used to read the reference physical time coordinates of each input data to generate a dynamic time window. It extracts the output data of the first-person visual understanding module, auditory understanding module, language recognition and medical translation module, medical terminology protection module and vital sign field whose reference physical time coordinates fall within the dynamic time window and whose round identifiers are consistent, performs association mapping, and aggregates and generates multimodal event clusters based on session identifiers and time window identifiers.

[0033] The evidence verification matrix module is used to instantiate a two-layer evidence verification matrix in memory. It performs multi-dimensional confidence calculation and cross-modal conflict detection with the derived clinical elements as the row dimension and the basic evidence layer and constraint verification layer as the column dimension. Based on the set modal confidence rules, it determines the support status of each data for the clinical elements and outputs clinical elements marked with high confidence, pending verification conflict or cross-modal conflict.

[0034] The Evidence-Based Medicine Intelligent Agent Adapter is used to extract relevant standard medical concept codes and patient historical interaction states into a request parameter package when receiving clinical elements marked as high confidence. It then calls the built-in evidence-based knowledge graph to initiate a query request, parses the returned knowledge graph path entities, and outputs a set of evidence-based context data containing associated target concept nodes and recommendation level weights.

[0035] The doctor operation guidance generation module integrates the conflict control signals output by the evidence mutual verification matrix module and the evidence-based context data set returned by the evidence-based medicine intelligent agent adapter. It combines the preset operation template to generate operation guidance actions for the current consultation round and formats the operation guidance actions into a structure containing action type, text prompt field and priority parameter.

[0036] The glasses-side limited display module is used to parse the priority parameters of the operation guidance actions, perform serialization and queuing, extract the text prompt field of the first-ranked operation guidance action and send it to the near-eye display component of the AI ​​glasses for visual rendering, while pushing the associated secondary evidence details into the display queue of the auxiliary terminal device.

[0037] The bilingual confirmation module is used to capture the voice, button or gesture feedback signals of the operator in response to the operation instructions through the AI ​​glasses. The feedback signals are parsed into confirmation records of adoption, modification, ignore or inapplicability, and the confirmation records are written back to the evidence mutual verification matrix module to cover the confirmation status value of the corresponding clinical elements.

[0038] See attached document Figure 2 Based on the above system architecture, this application provides a method for AI glasses audiovisual multilingual fusion diagnostic guidance, including the following steps: The S100 collects first-person perspective images and environmental audio signals through AI glasses. The first-person perspective images are input to the first-person perspective visual understanding module to extract visual features and object classification parameters. The environmental audio signals are input to the auditory understanding module to extract non-linguistic sound events. The environmental audio signals are input to the language recognition and medical translation module to determine the language, transcrib the speech to text, and perform cross-language semantic mapping. This generates the original text, translated text, source language identifier, timestamp parameters, medical terminology set, negation word set, symptom time expression, confidence level, and translation status. S200 extracts medically sensitive fields from the original text through the medical terminology protection module, generates independent protection identifiers for the medically sensitive fields, and blocks and isolates their confirmation status during the data flow process. S300 establishes a dynamic time window constraint based on the reference physical time coordinate through the cross-modal semantic alignment module. It then associates and binds the data output by the first-person visual understanding module, auditory understanding module, language recognition and medical translation module, and medical terminology protection module that fall within the dynamic time window and have consistent round identifiers, thereby generating a multimodal event cluster. S400 inputs multimodal event clusters into the evidence verification matrix module, constructs a two-layer evidence verification matrix to cross-compare data from different modal sources within the multimodal event clusters, performs multidimensional confidence calculations on clinical elements, and outputs status arbitration markers for high confidence, pending verification conflict, or cross-modal conflict. S500: When the evidence mutual verification matrix module outputs a highly credible labeling result, it triggers the evidence-based medicine intelligent agent adapter to call the built-in evidence-based knowledge graph to obtain the evidence-based context data set containing the associated target concept nodes and recommendation level weights. The doctor operation guidance generation module combines the conflict control signal output by the evidence mutual verification matrix module and the evidence-based context data set returned by the evidence-based medicine intelligent agent adapter to calculate and generate structured operation guidance actions through priority parameters. S600: The glasses-side limited display module serializes multiple operation guidance actions according to priority sorting rules and sends the first-ranked operation guidance action to the AI ​​glasses for visual presentation. The bilingual confirmation module collects feedback interaction signals through the AI ​​glasses to generate a confirmation record of the operation guidance action and sends the confirmation record back to the evidence mutual verification matrix module to perform reverse addressing routing calculation to update the confirmation status value of the corresponding clinical element.

[0039] To enable those skilled in the art to better understand this application, the following will elaborate on the technical details and specific implementation logic of the above-mentioned system operation and method execution process.

[0040] See attached document Figure 3 In step S100, for the visual information in the multimodal data, the system performs the acquisition and data encapsulation of first-view visual events. This process specifically includes the following sub-steps: The S101 continuously captures environmental video streams in clinical consultation, ward rounds, or outpatient scenarios using a built-in first-view camera on the AI ​​glasses. The system extracts frames from the environmental video stream at a preset frame rate, generating discrete first-view image sequences. During image extraction, the hardware clock component on the AI ​​glasses records the precise generation time of each frame, forming a timestamp. The first-view images and their corresponding timestamps are synchronously sent to the first-view visual understanding module in the processing computing node via a data transmission link.

[0041] S102, after receiving the first-view image sequence, the first-view visual understanding module initializes the visual analysis network in memory. The first-view visual understanding module parses the first-view images into a low-level computational structure, constructing a pixel matrix with the image's vertical spatial resolution as the row dimension, the image's horizontal spatial resolution as the column dimension, and color channel intensity values ​​as matrix elements. The visual analysis network performs parallel object detection and semantic segmentation operations, retrieving preset medical-related target objects from this pixel matrix. The preset medical-related target objects cover the patient's facial area, overall body shape, skin wounds and rashes, paper examination reports, drug packaging, bedside instrument reading panels, infusion bag labels, and surrounding environmental risk sources. For the convolutional layer feature extraction, bounding box regression, and classification probability calculation processes in the visual analysis network, those skilled in the art can use existing deep learning object detection algorithm architectures to implement them; their underlying network structure and training process are well-known technologies in the field and will not be elaborated upon here.

[0042] During target object extraction, the first-person perspective visual understanding module outputs the boundary coordinate matrix of the target object on the two-dimensional plane of the first-person perspective image based on the network calculation results, thereby generating the corresponding region of interest reference parameters. Simultaneously, the first-person perspective visual understanding module outputs the probability value of each target object's category through the classifier at the network's end, using this as the initial confidence parameter for the visual discovery result. For wounds and rashes, the first-person perspective visual understanding module further invokes morphological feature evaluation operators to calculate the area ratio, color deviation, and texture complexity of the lesion region. Based on the weighted summation of these parameters, a severity level parameter is mapped and output.

[0043] S103, the first-person perspective visual understanding module structurally combines the extracted target objects and their corresponding attribute parameters to generate discrete structured visual event data. This structured visual event data discards the triage logic used to directly assess the order of medical visits, abstracting it into a set of factual clues for subsequent system steps. The structured visual event data generated by the first-person perspective visual understanding module is represented by the following set model: ; in, Represents structured data entities for visual events; This represents the unique event sequence number assigned by the system to the current visual cue; This indicates the current round number and is used to define the time boundaries of the session context. This represents the category enumeration value of the target object, and its value range corresponds to preset recognition categories such as face, body shape, and wound. The region of interest reference parameter in the first-view image is used to indicate the spatial location of the target; This refers to the specific visual findings extracted from the region of interest. This indicates the severity level parameter calculated from the assessment. The confidence parameter for visual classification is calculated using a normalized exponential function, and its value is limited to the real number range of 0 to 1. This indicates a confirmation flag, which is a type of Boolean logic variable.

[0044] When the first-person visual understanding module determines that the current visual cue poses a clinical risk, and the confidence parameter... When the value falls below a preset safety tolerance threshold, the first-person perspective visual understanding module will need to verify that the marker position is in a true state. The specific value of this safety tolerance threshold can be set between 0.75 and 0.85 based on the clinical department's risk control standards. This verification of the marker position as true is used to indicate to the subsequent evidence verification matrix module that the visual cue needs to be cross-validated in conjunction with auditory data, text data, or human interaction records.

[0045] In step S100, for the environmental audio signals in the multimodal data, the system constructs an independent dual-track parallel parsing mechanism to achieve simultaneous extraction and structured processing of non-morpheme auditory signals and multilingual morpheme content. This process specifically includes the following sub-steps: The S104 continuously collects ambient audio signals in a clinical environment using a microphone array built into the AI ​​glasses. The system uses a speech endpoint detection algorithm to determine energy thresholds and truncate the continuous ambient audio signals, removing background silence segments that do not contain valid acoustic information to obtain discrete, valid audio segments. During audio segmentation, the system extracts the absolute start and end times of each audio segment using a hardware clock component, generating corresponding timestamp parameters. The acquired valid audio segments are transmitted to the processing-side computing node via a communication link. The processing-side computing node synchronously copies the same valid audio segment into two data streams, which are sent to the auditory understanding module and the language recognition and medical translation module, respectively, for parallel processing.

[0046] S105, after receiving a valid audio segment, the auditory understanding module converts the one-dimensional time-domain audio waveform data into a two-dimensional acoustic feature matrix. This acoustic feature matrix uses time frames as the row dimension and Mel-frequency cepstral coefficients as the column dimension. The auditory understanding module inputs the acoustic feature matrix into a built-in audio classification network to identify non-linguistic sound events contained in the valid audio segment. Non-linguistic sound events include a patient's cough, wheezing, vomiting, and alarms from bedside monitoring equipment. The construction and extraction of the acoustic feature matrix and the forward inference calculation process of the audio classification network can be implemented using existing acoustic event detection algorithm architectures by those skilled in the art. The underlying network structure and feature extraction process are well-known technologies in the field and will not be elaborated upon here.

[0047] The auditory understanding module extracts category parameters and corresponding classification confidence scores for non-morphemy sound events based on the output of the audio classification network. The module then combines the extracted acoustic elements to generate structured auditory event data. This structured data is represented by the following ensemble model: ; in, Represents structured data entities for auditory events; This represents the unique event sequence number assigned by the system to the current auditory cue; Indicates the current round number; An enumeration of non-morpheme sound event categories for classification determination; This represents the confidence output value of the audio classification network; This represents the timestamp parameter corresponding to the valid audio segment. When the confidence output value... When the auditory response exceeds the set threshold, the auditory understanding module pushes the structured data of the auditory event into the subsequent analysis queue. The specific value of this auditory response threshold is determined through field testing based on the background noise intensity of the clinical department, and its range is set between 0.80 and 0.90.

[0048] S106, the language recognition and medical translation module operates synchronously with the auditory understanding module, receiving valid audio segments and extracting their morphemes. The language recognition and medical translation module uses a language classification network to calculate the acoustic posterior probability of the source language to which the valid audio segment belongs, and determines the specific source language identifier of the input speech based on the maximum probability value. The language recognition and medical translation module loads the corresponding speech-to-text acoustic model and language model according to the source language identifier, decoding the valid audio segment into a discrete character sequence to generate the original text.

[0049] Based on the generated source text, the language identification and medical translation module invokes a cross-language semantic mapping model. This model obtains the system's preset target language configuration, converts the source text into the corresponding target language semantic expression, and outputs the translated text. To prevent the loss of clinical details in the original expression during cross-language mapping operations, the language identification and medical translation module does not directly overwrite the underlying data records with the translated text. Instead, it enforces a structured binding between the source and translated texts. The multilingual semantic data output by the language identification and medical translation module is represented by the following set model: ; in, Represents multilingual semantic data entities; This indicates the source language identifier determined by the aforementioned calculation; Indicates the target language identifier of the system; This represents the original text field generated from the transcription of a valid audio segment; This represents the translated text field generated through a cross-language semantic mapping model. This represents the timestamp parameter corresponding to the valid audio segment; This represents the set of extracted medical terms; This represents the set of negative words extracted. This represents the parsed symptom time expression parameters; A parameter representing the confidence level of the translation result; This indicates the current translation status, with values ​​including draft, need for back-translation confirmation, doctor's confirmation, and rejected.

[0050] This bilingual data encapsulation structure ensures that the system can directly extract the original text fields when semantic conflicts are detected during subsequent cross-modal reconciliation. By tracing the patient's original pronunciation, the risk of clinical intervention caused by translation errors from a single language model can be reduced.

[0051] See attached document Figure 4 In step S200, the system performs extraction operations on key information in the context that is likely to cause medical risks, and constructs an underlying state management structure for it. This process specifically includes the following sub-steps: S201, after receiving the multilingual semantic data entity output by the language identification and medical translation module, the medical terminology protection module extracts the original text field and the corresponding source language identifier. Based on the source language identifier, the medical terminology protection module further calls the system's pre-built medical-specific dictionary and rule base, and performs sequence labeling calculation with the original text field as the input source, anchoring and extracting medically sensitive fields from the character sequence.

[0052] The specific categories of extracted medically sensitive fields cover drug names, dosage parameters, allergy history records, and negative semantic expressions. For the character sequence segmentation, part-of-speech tagging, and named entity recognition forward inference operations in the original text fields, those skilled in the art can implement them using existing natural language processing algorithm architectures. The underlying sequence labeling network structure and dictionary matching logic are well-known technologies in the field and will not be elaborated upon here. This step forces the parsing of the original text rather than the translated text, ensuring that the obtained medically sensitive fields truly reflect the patient's original pronunciation intent, thereby cutting off the transmission path of key medical facts distorted due to translation bias caused by cross-language semantic mapping models.

[0053] S202, After extraction is complete, the medical terminology protection module generates an independent structured parameter container for each medically sensitive field instance and encapsulates these discrete text slices into a protected data set. The medically sensitive field protection entity generated in this process is represented by the following set model: ; in, This represents a system-generated entity protecting medically sensitive fields. This represents a globally unique protection identifier assigned to the currently sensitive field, which remains unchanged throughout the entire consultation session lifecycle; This represents the event sequence number inherited from the multilingual semantic data entity, which is used to retrieve the corresponding underlying valid audio segment for verification in subsequent processes. This indicates specific medically sensitive keywords extracted directly from the original text field; This parameter represents the enumeration of category labels to which the medically sensitive field belongs, and its value range corresponds to drug, dosage, allergy, or negative attribute; This indicates the candidate translations generated for this term; The confidence parameter represents the term extraction and translation. This indicates the current confirmation status value of the medically sensitive field, which is essentially a Boolean status register. This indicates a risk label associated with the medically sensitive field.

[0054] When generating entities for protecting sensitive medical fields, the medical terminology protection module uses hard-coded logic to include the confirmation status value. Initialize to an unconfirmed state. This unconfirmed state serves as a low-level latch flag, used to trigger data isolation rules during cross-modal alignment and fusion.

[0055] S203, for medically sensitive fields with a negative attribute classification label, the medical terminology protection module will simultaneously invoke the syntactic dependency parsing operator to perform context association calculation. This operator parses the syntax tree structure of the original text field and calculates the dependency path distance between the negative semantic expression words and the adjacent medical entity words. When the dependency path distance is less than the set dependency association threshold, the system will logically associate and bind the negative semantic expression words with the corresponding medical entities, merging and encapsulating them into the same medically sensitive field protection entity.

[0056] The specific value of the dependency association threshold is adaptively set based on the linguistic features corresponding to the source language identifier. For source languages ​​with standard subject-verb-object structures, the dependency association threshold can be set to an integer range of 1 to 3 tree node levels. By establishing syntactic binding relationships, the system can avoid incorrectly disassembling morpheme fragments such as "no history of penicillin allergy" in subsequent matrix comparisons, thus preventing the output of false positive clinical clues that contradict the patient's intended meaning.

[0057] In step S200, the system establishes a data blocking and isolation mechanism based on the aforementioned generated unconfirmed status identifier to prevent unverified sensitive information from flowing into the core decision-making process. This mechanism specifically includes the following sub-steps: S204, The system defines a set of access control rules for data flow in memory and links these rules with the confirmation status value in the medical sensitive field protection entity. A hard binding is implemented. This access control rule explicitly states that any medically sensitive field protection entity with an unconfirmed status value contains specific vocabulary content. It is prohibited as factual input during the data flow process and is provided to downstream modules used to generate final clinical elements or treatment recommendations.

[0058] Specifically, the downstream modules protected by this data blocking and isolation mechanism include: the core computing layer of the evidence mutual verification matrix module used to perform the final clinical element support status determination, the evidence-based medicine intelligent agent adapter used to encapsulate request parameters to call external knowledge systems, and any other business components used to automatically generate structured medical record drafts or clinical conclusions.

[0059] S205, in the system's data processing pipeline, when the data stream is passed from the medical terminology protection module to the subsequent cross-modal semantic alignment module and evidence verification matrix module, the system performs a pre-filtering and verification operation. This filtering and verification operation checks the confirmation status value of each medical sensitive field protection entity in the data stream one by one. .

[0060] If the check finds that the confirmation status is unconfirmed, the system will not directly discard the data entity, but will change its role and permissions in the data flow, specifically the lexical content. Logically shielded, it no longer participates in any form of semantic understanding or fact-inference calculations. The data entity becomes a pending confirmation item, and its only allowed downstream operation is to trigger an explicit interactive task aimed at confirming this field.

[0061] S206, these items carrying an unconfirmed status are guided to a dedicated, restricted logical path in the data flow diagram. They can enter a specific preprocessing area of ​​the evidence verification matrix module, but their role is limited to marking the presence of an unconfirmed information source for a certain clinical element in the matrix. The presence of this marker will directly trigger the physician operation guidance generation module, causing it to generate a high-priority operation guidance action of type back-translation confirmation or explicit follow-up questioning.

[0062] For example, when a drug name field with an unconfirmed status enters this pathway, the system generates an instruction prompting the physician to repeat or display the drug name to the patient and request confirmation. Until a positive interaction signal is received from the physician via the AI ​​glasses, the drug name field remains isolated; the system will not invoke the evidence-based medicine intelligent agent adapter to query potential drug interactions based on this unconfirmed information. This mechanism, by constructing a mandatory confirmation loop for medically sensitive fields, blocks the downstream transmission of inaccurate data caused by speech recognition errors or semantic mapping mistakes, preventing the system from generating incorrect instruction actions based on unverified information.

[0063] See attached document Figure 5 In step S300, the system performs cross-modal dynamic time window alignment. For asynchronous hash data from visual, auditory, and text channels, the system constructs a unified spatiotemporal reference system for multidimensional coordinate calibration. This process specifically includes the following sub-steps: S301, the cross-modal semantic alignment module concurrently receives various heterogeneous data entities output from the preceding processing pipeline. Specific data sources include structured visual event data output from the first-person visual understanding module, structured auditory event data output from the auditory understanding module, multilingual semantic data output from the language recognition and medical translation module, medical sensitive field protection entities output from the medical terminology protection module, and newly added vital sign fields and doctor confirmation status records. These data entities originate from independent computation channels and arrive in a non-continuous and asynchronous hash state when fed into the memory queue. Furthermore, all visual cues, auditory cues, translated semantics, and doctor confirmation information are required to be precisely bound using session identifiers, round identifiers, and event fragment identifiers to prevent contextual confusion caused by coarse-grained round binding.

[0064] In step S302, the cross-modal semantic alignment module traverses the received hash data entities and extracts the encapsulated time feature parameters. Specifically, it extracts the acquisition timestamps contained in the structured data of visual events, and the start times of the corresponding valid audio segments in the structured data of auditory events and the multilingual semantic data. To eliminate time deviations during the sampling process of heterogeneous sensors, the system maintains clock synchronization between the first-view camera and microphone array on the AI ​​glasses at the hardware level through a network time protocol. For the specific implementation of the device's underlying clock synchronization protocol, those skilled in the art can use existing network time synchronization protocol standards. The underlying network message interaction and hardware crystal oscillator compensation mechanisms are well-known technologies in the field and will not be elaborated upon here.

[0065] Based on hardware-level time synchronization, the extracted timestamp parameters are uniformly mapped to the system's absolute time axis. This cross-modal semantic alignment module assigns a reference physical time coordinate to each hashed data entity. This reference physical time coordinate It represents the precise absolute moment when a modal event occurs, measured in milliseconds (ms), and is used to provide a continuous physical time reference point in cross-modal alignment.

[0066] S303, based on the established physical time reference, the cross-modal semantic alignment module further analyzes the contextual state of clinical interactions, adding logical coordinate attributes to hash data entities. Combining the data output from the auditory understanding module and the language recognition and medical translation module, the system defines the consultation cycle of independent patients by recognizing auditory events with continuous long silent segments or the semantic text of consultation instructions issued by doctors, and generates a globally unique session identifier.

[0067] Within a defined conversation period, the pause intervals between the speakers are calculated using the energy endpoint states of the audio signals. When a question initiated by the doctor ends and the energy of the patient's response decays to the level of background noise, this span of voice interaction is defined as a complete consultation round. Based on this, round identifiers that increase in natural numbers are generated as logical time boundaries for dividing discrete consultation stages.

[0068] S304, After completing the calculation, the cross-modal semantic alignment module will combine the session identifier, round identifier, and reference physical time coordinates. These parameters are combined into a unified reference system and overwritten into the underlying structure of each hash data entity. Through this multi-dimensional coordinate calibration operation, a spatiotemporal reference system is established in memory space, with the session identifier as the primary key, the round identifier as the logical partition, and the reference physical time coordinate as the continuous scale.

[0069] Visual, auditory, and textual information, which were originally isolated on the timeline and logically independent, are forcibly correlated into the same coordinate system after the above calibration operation. The establishment of this reference system enables subsequent algorithm modules to overcome the asynchronous barrier of data flow and directly perform cross-modal cue truncation, extraction, and comparison calculations based on the same spatiotemporal metric.

[0070] In step S300, after completing the multidimensional coordinate calibration, the system performs dynamic time window constraint and alignment mapping calculations based on a unified spatiotemporal reference frame, aggregating discrete single-modal data into multimodal event clusters. This process specifically includes the following sub-steps: In step S305, after establishing a spatiotemporal reference system, the cross-modal semantic alignment module performs data filtering based on the same session identifier, loading hash data entities belonging to the current patient into computational memory. To link heterogeneous data expressing the same clinical semantics, the system introduces a dynamic time window constraint mechanism. Considering the significant differences in the interaction rhythm between doctors and patients during clinical consultations, using a fixed-width time window could easily lead to missed clues or mis-associations. Therefore, the system adopts a dynamic boundary delineation strategy based on semantic content.

[0071] Specifically, using multilingual semantic data entities or other high-confidence event entities as computational anchors, the baseline physical time coordinates within the multilingual semantic data entity of that anchor are extracted and denoted as... Based on this coordinate value, a backward time window threshold is set. With forward time window threshold Based on this, a closed time constraint interval is constructed. The backward time window threshold is used to capture visual or auditory cues that precede speech, such as a patient pointing towards a wound before expressing pain; its value is set between 1500ms and 3000ms. The forward time window threshold is used to capture response events that occur after a speech command is given; its value is set between 1000ms and 2000ms.

[0072] S306, based on the set dynamic time window interval, the cross-modal semantic alignment module traverses and searches the structured data of visual events, structured data of auditory events, medical sensitive field protected entities, and vital sign fields in memory. It extracts the baseline physical time coordinates of the aforementioned non-textual candidate target data entities and records them as... The system needs to satisfy logical round consistency, physical temporal proximity, and patient context consistency simultaneously to perform alignment mapping calculations. In addition, the system sets a maximum time span threshold. To avoid forcibly corroborating disparate anomalous clues in long question-and-answer scenarios, the specific value of this maximum time span threshold is set based on the average duration of a single clinical consultation topic, ranging from 10,000 ms to 30,000 ms. Data association must satisfy the following matching constraint expression: ; in, This indicates the matching result and is a boolean type. Indicates the round identifier of the candidate target data entity; A round identifier representing the multilingual semantic data entity used as an anchor point; This represents the logical AND operation; This indicates the maximum time span threshold. Patient context feature signature parameters representing candidate target data entities; This represents the patient context feature signature parameter of the data entity used as the anchor. When When the value is true, it is determined that the target data entity and the anchor data entity have a strong spatiotemporal semantic correlation and are merged into an alignment unit; if the two clues are in the same round but the time difference exceeds the maximum time span threshold, they will enter different dynamic time windows respectively.

[0073] S307, after completing the traversal and comparison, all data entities that meet the matching constraints are extracted and aggregated. These data, from different modalities but describing the same clinical interaction segment, are encapsulated into a structured multimodal event cluster. This operation breaks down the physical isolation of the original single-modal data, enabling cross-binding of underlying information. The multimodal event cluster is represented by the following set model: ; in, This represents the multimodal event cluster entity generated by aggregation; This is a unique identifier assigned by the system to this event cluster; This represents a session identifier used to define the current consultation period; Indicates the corresponding time window identifier; Representatives are multilingual semantic data entities that serve as the core anchor points; , , and Representing the reference physical time coordinates The set of structured data for visual events, the set of structured data for auditory events, the set of vital signs fields, and the set of entities protected by medical sensitive fields that fall within the dynamic time window and satisfy the round matching constraints.

[0074] If no data of a certain modality is matched within the time window, the corresponding set is set to empty. This does not affect the transmission of the entire event cluster entity to subsequent processing pipelines. Multiple generated multimodal event clusters are arranged chronologically and pushed into the evidence verification matrix module. Through the above calculations, the system transforms scattered feature parameters into composite data units with complete contextual information, laying the data structure foundation for subsequent logical reconciliation and factual inference.

[0075] See attached document Figure 6 In step S400, the system constructs a two-layer evidence mutual verification matrix to align and verify multimodal cues. This process specifically includes the following sub-steps: S401, the evidence verification matrix module receives multimodal event cluster entities transmitted from the upstream processing pipeline. It extracts feature attributes from the multilingual semantic data entities, visual event structured data sets, and auditory event structured data sets contained within the event clusters. Based on the extracted multimodal semantic content, it invokes the built-in medical logic reasoning network to perform forward inference, extracting core clinical elements from the current interaction segment. These clinical elements typically cover specific symptom presentations, abnormal signs, past medical history, or current medication status.

[0076] For the entity relation extraction and forward computation process of medical logic reasoning networks, those skilled in the art can implement it using existing graph neural network architectures or pre-trained language models. The underlying knowledge graph mapping and reasoning logic are well-known technologies in this field and will not be elaborated upon here. The derived set of clinical elements is denoted as... ,in The total number of clinical elements derived from the current multimodal event cluster is a positive integer. Each clinical element is represented by the following set model: ; in, A unique identifier representing this clinical element; Indicates the name of the original extracted feature; This indicates the source marker for the extrapolation of this clinical element; Indicates the normalized name after mapping; This indicates the risk label associated with the clinical element; This indicates the status value of the clinical element being confirmed.

[0077] S402, after acquiring the clinical element set, the evidence verification matrix module initializes the pre-verification matrix in memory space. Mutual verification matrix The matrix is ​​constructed using a row-column inverted semantic mapping structure, with the received visual cues, auditory cues, original semantics, translated semantics, and vital sign field channels as the row dimensions, and the set of clinical elements as the matrix dimensions. As the column dimension of the matrix, the column dimension of clinical elements is specifically formed by pre-set columns from the department template (covering symptoms, location, severity, medication, etc.) and newly added symptom extraction columns dynamically extracted in real time by the natural language processing module. Pre-verification matrix Used to record the consistency, missing, conflict, and low-confidence states of the original multimodal input, its main function is to truncate translations or ambiguous terms to be confirmed and trigger preliminary verification; while the confirmation mutual verification matrix It is specifically designed to carry clinical elements that have been confirmed by doctors, passed bilingual readback verification, and are consistent with high-confidence objective signs.

[0078] S403, for the initialized pre-mutual verification matrix Perform instantiation fill calculation. For the first... Clinical elements The support state of each modal channel is calculated, and the matrix elements are divided into discrete state sets of 1, 0, -1, and ?, corresponding to consistent support, missing evidence, cross-modal conflict, and low confidence states, respectively. Taking the visual evidence channel as an example, the structured data set of visual events is traversed to check for the existence of support states. This enumerates the categories of visual target objects that match the semantic mapping. If a high-match exists, it is assigned a value of 1 along with the extracted visual classification confidence score; if the visual discovery completely contradicts the auditory statement, it is assigned a value of -1; if the recognition confidence score is below a threshold, it is assigned a value of ?. Each matrix element simultaneously records source data citations, time window citations, confidence score citations, and cue basis citations to explain whether a specific conflict originates from underlying clinical evidence details such as the visual region of interest, the original text expression, or translation deviations.

[0079] S404: After populating the basic evidence layer with data, the constraint validation layer is instantiated simultaneously. This is for this clinical element. The timeliness and security of supporting evidence are assessed. The baseline physical time coordinates of the underlying data for each supporting modality are extracted, the absolute value of the time difference is calculated, and a spatiotemporal consistency score is calculated using a Gaussian decay function. Simultaneously, if the underlying traceability data of this clinical element is associated with the medically sensitive field protection entity generated in the aforementioned steps, its corresponding confirmation status value is extracted. Fill in the matrix. The state vector of the constraint verification layer is expressed as: ; in, Indicates the first Constraint verification vectors for each clinical element; This represents the calculated spatiotemporal consistency score, ranging from 0 to 1. The smaller the time difference, the closer the score is to 1. This represents a cross-language translation consistency flag, which is a Boolean type. This flag is obtained by the system backend calling a semantic similarity algorithm to compare the back-translated text of the source text field and the translated text field. It is used to indicate whether there is semantic loss in the reverse alignment operation between the source text field and the translated text field. When the similarity is lower than the set threshold, it is set to false; otherwise, it is set to true. This represents the confirmation status value of the associated medical sensitive field. It is a Boolean value that is directly inherited from the corresponding medical sensitive field protection entity.

[0080] Through the above filling operations, the complete record of the two-layer evidence mutual verification matrix for specific clinical elements in the row dimension is expressed as follows: This instantiated matrix structure rigorously encapsulates the basic confidence vector describing the facts of the illness and the constraint state vector that determines whether the facts can be accepted, providing a standardized computational foundation for subsequent conflict detection and logical arbitration.

[0081] In step S400, after instantiating the two-layer evidence mutual verification matrix, the system performs multi-dimensional confidence calculation based on the state vector encapsulated within the matrix, and uses the calculation results to drive a cross-modal conflict arbitration mechanism to determine the final acceptance status of clinical elements. This process specifically includes the following sub-steps: S405, the evidence verification matrix module processes the pre-verification matrix. When encountering conflicting or low-confidence (values ​​of -1 or ?) nodes, a simple fixed-weight linear weighting is no longer used. Instead, a modal confidence verification mechanism is introduced for quantitative evaluation. The system calculates the modal confidence score for a specific modal source, and the calculation model is represented by the following formula: ; in, This represents the calculated modal confidence score; This indicates the objective reliability of the modality source (e.g., the reliability of objective vital signs instruments is higher than that of subjective translation records). This represents the raw confidence level output by the underlying algorithm model; Indicates sensor acquisition quality assessment parameters (such as the sharpness of the visual region of interest); This represents the matching scaling factor for the dynamic time window. This multiplicative model ensures that any single-dimensional defect (such as acquisition ambiguity or temporal misalignment) will lead to a decrease in the credibility of the modal cue.

[0082] S406, after calculating the modal credibility score of each source, the evidence verification matrix module executes priority-based state arbitration logic. When cross-modal contradictions occur (e.g., visual extraction of suspected rashes but the patient's original text explicitly denies it through translation), the system compares the modal credibility scores of both parties: if the visual source is due to... Low or If the overall score is lower than the translated source, the system arbitration generates an action instruction to "examine the skin area or reshoot." If the visual source has a high confidence score, but the denial statement originates from a low-confidence draft-level translation, the system prioritizes generating a "back-translation confirmation" action. For the pre-verification matrix... For elements containing low-confidence translations, negative word conflicts, or terminological ambiguities, the system only allows triggering interactive commands such as confirmation and follow-up questions, without directly identifying a conflict. Only after receiving confirmation from a doctor or being determined to be a high-risk objective sign abnormality are the relevant elements allowed to be written into the confirmation matrix. .

[0083] S407, for the different types of arbitration marker results mentioned above, the evidence verification matrix module performs a split output operation. The system extracts the modality confidence scores of all non-conflict modalities under this clinical element. A weighted aggregation calculation is performed to obtain the overall confidence level of this clinical element. .when Greater than or equal to the overall confidence threshold (This threshold is set to a real number between 0.80 and 0.95 depending on the system's strictness.) If no conflict or blocking flag is triggered, the system marks the clinical element as having high reliability and sends it to the evidence-based medicine intelligent agent adapter and the structured medical record generation pipeline as a reliable prior fact for subsequent auxiliary diagnosis and treatment decisions.

[0084] For clinical elements marked as pending conflict or cross-modal conflict, the evidence verification matrix module synchronously transmits control signals and the element's underlying data source to the doctor's operation guidance generation module. Upon receiving the control signals, the doctor's operation guidance generation module immediately blocks the element from participating in internal automated diagnostic deduction. Based on the recorded conflict type and missing modal evidence, the doctor's operation guidance generation module dynamically assembles interactive prompts and issues interventional operation guidance to the doctor, such as "Please repeat and confirm the drug dosage manually" or "The patient's verbal description of pain does not match the physical signs; please perform a palpation examination." Through this arbitration mechanism, the system establishes a closed loop for a smooth transition from algorithmic uncertainty to human decision-making, effectively preventing the clinical risk of single-modal recognition errors spreading outward.

[0085] See attached document Figure 7 In step S500, the system invokes the evidence-based knowledge base to match reliable disease clues with medical guidelines, thereby providing logical support for the generation of subsequent guidance actions. This process specifically includes the following sub-steps: S501, the evidence-based medicine intelligent agent adapter receives confirmation record data output by the evidence verification matrix module. The system establishes a mandatory filtering barrier, stipulating that the evidence-based medicine intelligent agent is only allowed to call the confirmation verification matrix. The system initiates searches for clinical elements that exist in the database and are marked as high-risk for enhanced consistency, key field conflicts, or abnormal objective signs, while rejecting all translated fields in an unconfirmed state (draft stage) to prevent false diagnostic suggestions. To eliminate the interference of the variability of natural language expressions on knowledge retrieval, the evidence-based medicine intelligent agent adapter performs medical concept normalization processing on the input clinical elements. In the specific execution process, the evidence-based medicine intelligent agent adapter calls the system's built-in medical standard terminology mapping dictionary to map the free text descriptions in the clinical elements to unique standard medical concept codes. The above mapping operation converts the free text into independent node identifiers in the standard disease classification system or clinical medical terminology set.

[0086] For the specific string matching and semantic similarity calculation process of medical term mapping, those skilled in the art can use existing dictionary-based entity linking algorithms or pre-trained language models to implement it. The underlying inverted index establishment and text vectorization operation are well-known technologies in the field and will not be elaborated here.

[0087] S502, after completing the medical concept normalization, the evidence-based medicine intelligent agent adapter performs a state-driven query encapsulation operation based on the patient's current dynamic interaction context. The adapter extracts the current patient's historical confirmed state records using session identifiers from the previously generated spatiotemporal reference frame. These past facts confirmed by the two-layer evidence verification matrix (e.g., the patient's verified chronic disease history or current accurate medication records) are used as mandatory constraints for the graph query. This state-driven query request entity is represented by the following set model: ; in, This represents the encapsulated and generated query request entity; This is a globally unique serial number assigned to this query task; This represents the normalized standard medical concept code, serving as the starting core node for knowledge retrieval; This represents the set of query constraints derived from the patient's current historical interaction state; This represents the depth limit threshold for graph network traversal. This depth limit threshold determines the number of hops the system needs to make when expanding outwards in the graph database to search for relevant disease or drug features. Its value is set to an integer from 1 to 3 to achieve a balance between retrieval logic recall and system response latency.

[0088] S503, the evidence-based medicine intelligent agent adapter will construct the query request entity. The query is converted into a standard query protocol statement recognizable by the underlying graph database and a network call request is initiated to the system's built-in evidence-based knowledge graph. Upon receiving the call request, the evidence-based knowledge graph uses the input starting core node... With depth limit threshold Perform graph traversal calculations. During the traversal and pathfinding process, the system's underlying search engine verifies in real time whether path nodes satisfy the set of query constraints. The constraints in the code are as follows. If a logical extension path is found to violate constraints such as drug contraindications, that retrieval branch is immediately pruned and discarded. After constraint state filtering, the evidence-based knowledge graph returns valid medical logical association paths. These returned knowledge graph path entities are represented as follows: ; in, This represents the single valid knowledge graph path entity returned. and These represent the starting concept node of the path and the derived associated target concept node, respectively. This represents an enumeration of medical logical relationships connecting two nodes, specifically covering relationship categories such as causing symptoms, belonging to complications, and applicable to specific physical examination procedures; This indicates the recommendation level weight of the evidence-based pathway in clinical guidelines. The value range is mapped to a real number interval from 0 to 1. The closer the value is to 1, the stronger the authoritative guidance of the associated medical fact in clinical practice.

[0089] S504, the evidence-based medicine intelligent agent adapter parses the various knowledge graph path entities returned, and extracts the associated target concept nodes. With corresponding recommendation level weights After extraction, the evidence-based medicine intelligent agent adapter temporarily stores these filtered medical knowledge points in a memory pool, forming an evidence-based context data set for the current highly trusted clinical elements. Through this dynamic invocation mechanism based on state constraints, the system avoids inputting redundant or conflicting broad knowledge into subsequent calculation stages, ensuring that the extracted evidence matches the specific pathological context of a particular patient, thus laying the underlying knowledge structure foundation for the system to output accurate intervention actions.

[0090] In step S500, after acquiring the corresponding conflict control signal or evidence-based context data, the system performs structured encapsulation calculations for operation guidance oriented towards diagnosis and treatment actions, generating specific intervention instructions which are then sent to the terminal. This process specifically includes the following sub-steps: In S505, the doctor's operation guidance generation module extracts two types of heterogeneous driving source data from the system's memory queue. One type of data is the conflict control signal output by the evidence mutual verification matrix module during the conflict arbitration phase. This signal contains clinical elements marked as pending conflict or cross-modal conflict and their missing modal evidence descriptions. The other type of data is the evidence-based medicine intelligent agent adapter's set of evidence-based context data retrieved and temporarily stored based on highly reliable clinical elements, containing associated target concept nodes and recommendation level weights. The doctor's operation guidance generation module integrates these two types of data into its internal processing pipeline for synchronous calculation.

[0091] S506, the doctor's operation guidance generation module, in conjunction with a preset operation template, calculates a priority parameter for each input driver source data and determines the action type of the operation guidance. The doctor's operation guidance generation module defines the priority parameter as follows: Its value ranges from 0 to 1, and the larger the value, the higher the urgency of the intervention in clinical practice.

[0092] The allocation of guided action types is based on the risk assessment and status of the mutual verification matrix: when the driving data comes from the pre-mutual verification matrix. Furthermore, when the translation contains low-confidence or ambiguous terminology, the system will forcibly assign the action type of back-translation confirmation or requesting a doctor to repeat the translation for confirmation in order to obtain the accurate meaning of the original text; when the driving data comes from the confirmation and verification matrix... It also reflects the priority parameters when facing objectively high-risk conditions (such as decreased blood oxygen levels coupled with environmental alarms). The data is elevated to the highest level and assigned the action type as emergency observation or follow-up questioning and physical examination. For stable data in the evidence-based context dataset, the system determines it belongs to the auxiliary decision-making suggestion level. The doctor's operation guidance generation module extracts the built-in recommendation level weights of this data and maps them to specific priority parameters. For this type of data, the doctor's operation guidance generation module uniformly assigns the action type enumeration values ​​as treatment recommendations, which are used to prompt operators to pay attention to specific medical indicators.

[0093] S507, after completing the priority parameter calculation and action type determination, the doctor's operation guidance generation module performs formatted assembly calculation of the execution information, packages the hashed calculation results, and generates a structured operation guidance action. This operation guidance action is represented by the following data model: ; in, This represents the structure containing the formatted operation instructions. The globally unique sequence number assigned by the system to this instruction; Indicates the session identifier; Indicates the round number; Indicates the time window identifier; This represents the action type enumeration value calculated above; This refers to the actual text displayed in the AI ​​glasses' limited-width rendering; This indicates the details of the supporting evidence cited to support the generation of this action; A marker indicating whether the action requires mandatory confirmation from a doctor; This indicates the current doctor status of the action (it can be initially set to unprocessed, and subsequently receive records of adoption, modification, etc.). For the corresponding priority parameter; A text prompt field for operators to read; This indicates the specific reason for assigning the current priority parameter. The doctor's operation guidance generation module has multiple built-in natural language operation templates. By filling parameters such as missing evidence descriptions or related target concept nodes into the placeholders of the corresponding templates, the string data of the text prompt field is generated.

[0094] S508, the doctor's operation guidance generation module will generate a series of operation guidance actions. Push in the glasses-side limited display module. To prevent information overload from interfering with the normal consultation rhythm, the glasses-side limited display module follows the priority parameters contained within each operation guide action. Execute serialized queuing scheduling. The glasses-side limiting display module establishes a priority-based descending queue, and high-priority actions are dynamically inserted at the head of the queue.

[0095] After sorting, the eyewear-side limited-amplitude display module extracts the top-ranked operational guidance action, parses and extracts the text prompt field from this action, and sends it as a text prompt field to the AI ​​eyewear via a wireless communication link. Upon receiving the data, the AI ​​eyewear drives its built-in near-eye display component to visually render and present the text prompt field. Simultaneously, the eyewear-side limited-amplitude display module pushes the secondary evidence details associated with this operational guidance action into the display queue of the auxiliary terminal device. Through this structured encapsulation and scheduling mechanism, the system transforms the complex backend calculations into intuitive and orderly interventional operational guidance from the operator's first-person perspective.

[0096] See attached document Figure 8 In step S600, the system performs edge-side amplitude limiting scheduling and visual presentation control for the generated operation guidance actions. This process specifically includes the following sub-steps: S601, the glasses-side limited display module receives multiple operation guidance actions generated by the preceding processing pipeline. In clinical consultation scenarios, doctors need to maintain visual focus on the patient, but the display area and available field of view of the near-eye display component built into the AI ​​glasses have physical limits. To prevent instantaneous information overload from interfering with the diagnostic rhythm, the glasses-side limited display module establishes a concurrency constraint mechanism for the display channel, performing memory buffering and state isolation on the incoming operation guidance action stream.

[0097] S602, the eyeglass-end limiting display module serializes multiple operation guidance actions in the buffer queue according to a priority sorting rule. The eyeglass-end limiting display module extracts the priority parameters within each operation guidance action. and the system timestamp corresponding to when the action was generated. The system uses the above parameters to perform a descending sorting calculation on the elements in the buffer queue, outputting a decreasing ordered instruction queue. This ordered instruction queue is represented by the following set expression: ; in, This represents an ordered instruction queue after serialization and queuing. This represents the total number of backlogged operation guidance actions in the current queue, and is a positive integer. Indicates ranking in The operation guide for each bit. The positions of elements in the queue satisfy the following sorting constraints: ; When the priority parameters of two adjacent elements are equal, that is At that time, the system introduces auxiliary sorting constraints based on the order of generation time: ; S603, after completing the serialization process, the eyeglass-side limiting display module performs truncation control. The eyeglass-side limiting display module has a built-in near-eye limiting threshold. This is used to limit the number of independent tasks allowed to be rendered on the glasses at the same time. This near-eye limiting threshold takes into account the visual load boundary of the monocular field of view. The default setting is 1.

[0098] Based on this near-eye limiting threshold setting, the eyeglass-side limiting display module extracts the first-ranked operation guidance action. The action structure is parsed, and the text prompt field is extracted. The limited display module on the glasses sends the extracted text prompt field to the AI ​​glasses via a wireless communication link. After receiving the data, the AI ​​glasses drive the near-eye display component to render and overlay the text prompt field onto the doctor's first-person view. For the optical rendering and layout engine of the near-eye display component, those skilled in the art can use existing augmented reality display control frameworks. Its underlying micro-projection control and frame buffer refresh mechanism are well-known technologies in the field and will not be elaborated upon here.

[0099] S604: While sending the top-ranked instruction to the AI ​​glasses for visual presentation, the glasses' amplitude-limiting display module performs downgraded output processing for the remaining elements in the queue. (The set of instruction sets ranked after the near-eye amplitude-limiting threshold.) They will not be discarded. The glasses-side limited display module extracts the secondary evidence details and auxiliary reference guidelines associated with these degradation actions and redirects them into the display queue of the assistive terminal device. The assistive terminal device is typically a desktop monitor or portable tablet computer in the examination room.

[0100] Through this limiting scheduling and degraded traffic calculation, the system ensures that doctors can obtain the highest priority conflict verification instructions with zero latency through the AI ​​glasses, while seamlessly migrating non-urgent treatment recommendations to the external screen. This mechanism maintains data balance between the primary and secondary display channels, avoiding the omission of critical medical intervention information.

[0101] In step S600, after rendering the terminal guidance interface, the system captures the doctor's feedback and converts it into update instructions for the underlying data state, thus forming a closed loop. This process specifically includes the following sub-steps: S605, once the text prompt for the first operation instruction is visually presented on the AI ​​glasses, the bilingual confirmation module simultaneously activates the input acquisition channel on the AI ​​glasses. The bilingual confirmation module continuously listens for and acquires the feedback interaction signals emitted by the operator in response to the current prompt information. These feedback interaction signals cover multiple modalities of physical input, specifically including voice feedback signals acquired through the microphone array, button level signals captured through the device's side touchpad, and specific gesture video streams acquired through the first-person view camera.

[0102] For the intent classification of voice commands, anti-shake filtering of touch buttons, and spatial trajectory recognition of gestures, those skilled in the art can use existing acoustic models, digital signal processing frameworks, or three-dimensional convolutional neural networks to achieve these. The underlying feature extraction and pattern matching calculations are well-known technologies in the field and will not be elaborated here.

[0103] S606, the bilingual confirmation module performs intent classification calculations on the extracted feedback interaction signal features, parsing the unstructured physical signals into standard confirmation records. These confirmation records not only carry the operator's decision results but also contain index identifiers for tracing underlying data. The generated confirmation records are represented by the following set model: ; in, This indicates the confirmation record generated during the encapsulation process; The globally unique sequence number representing the operation guidance action being responded to is temporarily stored by the system during rendering and retrieved at this moment; This represents the parsed feedback operation enumeration value, which the system limits to four discrete states: adopted, modified, ignored, or not applicable. This field represents the new corrected numerical parameter that the system synchronously captures when the operator triggers a modification command, such as the doctor verbally stating the changed dosage parameter "10mg". If the feedback operation is determined to be adopted or ignored, this field is set to empty.

[0104] S607, the bilingual confirmation module will build the confirmation record. The evidence verification matrix module is transmitted back to the processing side via the uplink communication link. Upon receiving the confirmation record, the evidence verification matrix module performs reverse addressing routing calculations and extracts the relevant information from the confirmation record. As a source tracing index parameter, the context chain at the time the operation guidance action was generated is retrieved in reverse from the system's memory log space. Through index matching, the evidence mutual verification matrix module accurately locates the underlying multimodal event cluster that triggered the intervention action and extracts the corresponding first... Clinical elements and its corresponding constraint verification vector in the matrix .

[0105] S608, after successful addressing, the evidence verification matrix module verifies the information in the confirmation record. Update the confirmation status value of the corresponding clinical element. The system internally establishes a set of state coverage logic branches: when When the value is "accepted," it indicates that the operator has acknowledged the data captured by the system or performed the suggested verification action. The evidence verification matrix module directly assigns the confirmation status value from the constraint verification vector. The value is overwritten by the false (or unconfirmed) state and assigned to true.

[0106] when When the value is modified, it indicates that there is a deviation in the element parameters previously captured by the system, and the evidence mutual verification matrix module extracts... The data in the data is used to replace the corresponding erroneous entity attributes in the basic evidence vector, and then the confirmation status value is synchronously transmitted. The overwrite assignment is true.

[0107] when When the value is ignored or inapplicable, it indicates that the operator has determined that the prompt is invalid or that the current scenario does not meet the conditions for execution. The system marks the clinical element as obsolete and removes it from the matrix calculation queue.

[0108] With the confirmation status value of the corresponding clinical elements Once updated to true, the system's previous mandatory blocking mechanism for this element is lifted. The evidence verification matrix module recalculates the multidimensional comprehensive confidence level of the clinical element based on the updated state. After the state is closed, the clinical element is allowed to flow into the subsequent data aggregation pipeline. The system thus completes the entire chain calculation from clue capture, conflict detection, end-side intervention to manual state confirmation.

[0109] Specific application examples: To aid in understanding the technical solution of the AI ​​glasses audiovisual multilingual fusion consultation guidance system and method of this application, a specific application example is given below in the context of emergency consultation for drug allergies in foreign patients.

[0110] During the data collection and low-level extraction phase, a foreign patient whose native language is English was admitted to the emergency room with severe erythema on their arm and a mild cough. The doctor received the patient while wearing the AI ​​glasses described in this application. The first-person view camera on the AI ​​glasses captured an image of the patient's arm, and the first-person view visual understanding module extracted the erythema area and calculated the severity level parameters. The confidence level was relatively high, but due to the lighting conditions in the emergency room, the confidence parameters for classification were affected. The value is 0.72 (below the preset safety tolerance threshold of 0.75). Therefore, the visual event data is structured. Confirmation flag in Set to true.

[0111] During the auditory extraction phase, a microphone array records ambient sound. The auditory understanding module identifies non-morphemymal sound events and classifies them into categories. For "coughing sound", the confidence output value is... It is 0.88.

[0112] During the text extraction phase, the patient verbally stated, "I took Penicillin, but the rash got worse." The language identification and medical translation module determined the source language identifier. For English, the target language for the job is indicated. For Chinese. Output multilingual semantic data, where the original text field is in Chinese. Translate the text field for "I took Penicillin…". The patient wrote, "I took penicillin, but my rash worsened."

[0113] During the medical sensitive field blocking and isolation phase, the medical terminology protection module performs sequence annotation on the original text, accurately extracts "Penicillin," and then generates a medical sensitive field protection entity. Its category label enumeration parameters For drugs, specific medically sensitive terms The word is "penicillin". At this point, the system initializes its confirmed status value to an unconfirmed state through hard-coded logic, thereby triggering a data blocking and isolation mechanism to prevent this specific word from directly entering downstream modules during data flow.

[0114] During the cross-modal dynamic time window alignment stage, the cross-modal semantic alignment module uses multilingual semantic data as anchor points to extract the baseline physical time coordinates. Set the backward time window threshold The forward time window threshold is 2000ms. The time limit is 1000ms. After traversing and calculating, the baseline physical time coordinates of the visually detected erythema and the auditory extracted cough sound all fall within this closed time constraint interval, and the time span meets the maximum time span threshold set by the system. All data satisfies the logical AND operation. They are aggregated and encapsulated into the same multimodal event cluster entity. .

[0115] During the mutual evidence matrix calculation and arbitration phase, the evidence mutual evidence matrix module instantiates a two-layer evidence mutual evidence matrix. This is for the first clinical element derived from the derivation. (Suspected penicillin allergy), pre-verification matrix The confirmation status value of the term "penicillin" was found to support the facts. Unconfirmed.

[0116] System calculation modal reliability score Although the translation confidence level is acceptable, the system outputs a status arbitration flag indicating a pending conflict because the translation is in an unconfirmed state.

[0117] During the evidence-based interception and guidance action generation phase, due to Not marked as high-confidence, the evidence-based medicine intelligent agent adapter was intercepted, preventing the invocation of the built-in evidence-based knowledge graph to avoid providing incorrect medication recommendations. The doctor's operation guidance generation module received a conflict control signal and generated operation guidance actions based on a preset operation template. Action type enumeration value Assigned as back-translation confirmation, priority parameter The highest value of 0.95 is calculated for the text prompt field. The generated message is "Please verify the drug: Penicillin".

[0118] During the narrowing display scheduling and interaction closed-loop phase, the eyewear-side narrowing display module has the highest extraction priority. Actions are arranged in an ordered instruction queue. First. Based on the near-eye limiting threshold set to 1 by default. The message "Please verify medication: penicillin" is rendered separately and overlaid on the doctor's first-person view.

[0119] After seeing the prompt, the doctor repeated "Penicillin?" to the patient, who nodded. The doctor then lightly touched the touchpad on the side of the glasses. The bilingual confirmation module generated a confirmation record. Feedback operation enumeration value The data is resolved and accepted. The data is then sent back to the evidence verification matrix module, where the corresponding constraint verification vector is retrieved. Confirm its status value The unconfirmed state is overwritten with a value of true. The mandatory blocking mechanism is then lifted, and the system recalculates the high-confidence multidimensional comprehensive confidence level based on the updated state. It is authorized to flow into the subsequent data aggregation pipeline, completing the information loop.

[0120] Regarding the above embodiments and the underlying computational logic of the system, the following experimental verification and effect comparison are provided in conjunction with the accompanying drawings.

[0121] See attached document Figure 9 The figure shows how the modal credibility score changes with the time difference of audiovisual cues when the evidence mutual verification matrix module performs multidimensional confidence calculation. The horizontal axis represents the absolute time difference between the baseline physical time coordinates of the target data entity and the anchor data entity, in milliseconds (ms), and the vertical axis represents the modal credibility score.

[0122] from Figure 9 As can be seen from the curve trend, the modal confidence score gradually decreases as the time difference increases. When the absolute time difference is 0, it indicates that the multimodal cues occur at the same time, and the modal confidence score is at its highest value, approximately 0.77. As the absolute time difference continues to increase, the modal confidence score exhibits a non-linear decreasing trend, and when the absolute time difference reaches approximately 2500 ms, the score drops below 0.2.

[0123] The reason for these changes is that the system introduces a matching degree scaling factor for a dynamic time window, which is affected by the absolute time difference. The greater the deviation of the multimodal cues acquired by the sensor in physical time, the lower the spatiotemporal consistency, and the lower the modal reliability score calculated by weighted calculation. This avoids the problem of hard truncation caused by small time differences in the traditional fixed time window method.

[0124] See attached document Figure 10 This figure illustrates the comparison of the multidimensional comprehensive confidence scores of three typical clinical elements in the system before and after the loop closure is completed. The horizontal axis represents the three specific clinical elements extracted from the system, which are: clinical elements... Clinical elements With clinical elements The vertical axis represents the multidimensional comprehensive confidence level calculated by the system for clinical elements.

[0125] Combination Figure 10 The values ​​can show the changes in various clinical elements. Because the associated medically sensitive field was initially assigned an unconfirmed status, a blocking and isolation mechanism was triggered, and its initial multidimensional comprehensive confidence level before the loop closure was only 0.02; after the system received the confirmation record of the adoption action and the status was updated to true, its multidimensional comprehensive confidence level increased to 0.95. Clinical elements The visual classification confidence level was low due to light interference, and the system was in a pending verification state before loop closure, with a multidimensional composite confidence level of 0.65. After interactively obtaining the operator's corrected values, the multidimensional composite confidence level was updated to 0.88. (Clinical elements) The data was read by objective vital signs instruments and there were no cross-modal conflicts. The initial multidimensional composite confidence level reached 0.92, which was marked as high confidence level. No manual intervention was required, so the confidence level remained unchanged before and after the loop closure.

[0126] The reason for this difference lies in the fact that the system is configured with computational logic based on feedback operation status coverage. The system can replace the attributes and confirmation status values ​​of clinical elements according to the manual confirmation records collected by the device, remove the blockage and recalculate the confidence level, thereby realizing closed-loop control from the initial algorithm calculation to manual intervention confirmation.

[0127] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0128] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0129] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for providing multilingual audiovisual consultation guidance using AI glasses, characterized in that: Includes the following steps: First-person view images and environmental audio signals are acquired to generate structured data of visual events, structured data of auditory events, and multilingual semantic data; Extract medically sensitive fields from the multilingual semantic data to generate a medically sensitive field protection entity; The structured data of visual events, structured data of auditory events, multilingual semantic data, and the entities protecting medical sensitive fields that satisfy the dynamic time window constraints are aggregated to generate a multimodal event cluster. The multimodal event clusters are input into a two-layer evidence verification matrix to deduce clinical elements, and clinical elements or conflict control signals with high confidence acceptance labels are output. The clinical elements with high confidence acceptance markers are used to obtain an evidence-based context data set, and the conflict control signal and the evidence-based context data set are combined to generate operational guidance actions. The operation guidance actions are sent to the AI ​​glasses for display, feedback interaction signals are collected to generate a confirmation record, and the confirmation status value of the clinical element is updated based on the confirmation record.

2. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 1, characterized in that, The generated structured data for visual events includes: Target detection and semantic segmentation are performed on the first viewpoint image. The boundary coordinate matrix of the target object is extracted to generate the region of interest reference parameter. The probability value of the target object's category is output as the confidence parameter. Call the morphological feature evaluation operator to map and output the severity level parameter; The target object, the region of interest reference parameter, the confidence parameter, and the severity level parameter are combined to generate structured data of the visual event. When the confidence parameter is lower than the safety tolerance threshold, the position of the confirmation flag is set to true.

3. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 1, characterized in that, Generate the structured data and multilingual semantic data of the auditory events, including: The environmental audio signal is truncated and segmented to obtain effective audio segments. The effective audio segments are converted into an acoustic feature matrix, input into an audio classification network to identify non-morpheme sound events, and combined with the extracted timestamp parameters to generate auditory event structured data. The effective audio segment is input into a language classification network to determine the source language identifier, a language model is loaded to decode and generate the original text, a cross-language semantic mapping model is called to convert the original text into translated text, and the source language identifier, the original text and the translated text are structurally bound to generate multilingual semantic data.

4. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 3, characterized in that, Extracting medically sensitive fields from the multilingual semantic data to generate a medically sensitive field protection entity includes: The medical-specific dictionary and rule base of the language are called to perform sequence labeling calculation on the original text in the multilingual semantic data to extract medical sensitive fields, and encapsulate and generate a medical sensitive field protection entity containing a confirmation status value, and initialize the confirmation status value to an unconfirmed state; Check the confirmation status value of the medical sensitive field protection entity. For medical sensitive field protection entities in the unconfirmed state, block specific word content from participating in semantic understanding and fact inference calculations, and trigger an explicit interactive task to confirm the specific word content.

5. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 4, characterized in that, The structured data of visual events, structured data of auditory events, multilingual semantic data, and the entities protected by the medical sensitive fields, which satisfy the dynamic time window constraints, are aggregated to generate a multimodal event cluster, including: Extract the temporal feature parameters of the visual event structured data, auditory event structured data, and multilingual semantic data, map them to an absolute time axis, and assign them a reference physical time coordinate; Identify the environmental audio signals to define the consultation cycle and generate a session identifier, and define the consultation rounds and generate round identifiers; The dynamic time window constraint is formed by setting forward time window thresholds and backward time window thresholds using the reference physical time coordinates. Data that fall within the dynamic time window constraint and meet the conditions of consistent round identifiers, time span not exceeding the maximum time span threshold, and consistent patient context feature signature parameters are associated and mapped to generate multimodal event clusters.

6. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 5, characterized in that, The multimodal event clusters are input into a two-layer evidence verification matrix to deduce clinical elements, and clinical elements or conflict control signals with high confidence acceptance markers are output, including: The initialization of the pre-mutual verification matrix and the confirmation mutual verification matrix serve as the two-layer evidence mutual verification matrix; The medical logic reasoning network is invoked to extract clinical elements from the multimodal event cluster, calculate the support status of different modal channels for the clinical elements, and fill the basic evidence layer into the pre-mutual verification matrix. Assess the time compactness and state security corresponding to the support state, calculate the spatiotemporal consistency score, and fill the constraint verification layer of the pre-mutual verification matrix together with the cross-language translation consistency flag and the confirmation state value of the medical sensitive field protection entity. Based on the support status, multidimensional confidence calculation is performed on the clinical elements. For clinical elements whose confidence reaches the comprehensive confidence threshold, a high confidence acceptance flag is output, and for clinical elements containing conflicts, a conflict control signal is output.

7. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 6, characterized in that, Using the clinical elements with high confidence markers, an evidence-based contextual data set is obtained. Combining the conflict control signals and the evidence-based contextual data set, operational guidance actions are generated, including: The clinical elements with high confidence acceptance tags are subjected to medical concept normalization processing and mapped to standard medical concept codes; Extract the current patient's historical confirmation status records as a set of query constraints, and combine them with the standard medical concept encoding and depth limit threshold to encapsulate and generate a query request entity; The query request entity is used to initiate a call to the built-in evidence-based knowledge graph. Based on the standard medical concept encoding and depth limit threshold in the query request entity, graph traversal calculation is performed. Logical extension path pruning and filtering are performed according to the query constraint set to obtain the knowledge graph path entity containing associated target concept nodes and recommendation level weights as the evidence-based context data set. The priority parameters and action type enumeration values ​​are calculated based on the received conflict control signal and the evidence-based context data set, combined with a preset operation template. Fill the placeholders in the operation template with missing evidence descriptions or related target concept nodes to generate a structured operation guidance action that includes text prompt fields, action type enumeration values, and priority parameters.

8. The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method according to claim 7, characterized in that, The operation guidance actions are sent to the AI ​​glasses for display, feedback interaction signals are collected to generate a confirmation record, and the confirmation status value of the clinical element is updated based on the confirmation record, including: The priority parameters in the operation guidance actions are extracted, sorted in descending order, and serialized to output an ordered instruction queue. The text prompt field of the operation guidance action ranked first in the ordered instruction queue is sent to the AI ​​glasses for rendering and overlay. The system collects feedback signals from operators, classifies and calculates their execution intentions, and generates a confirmation record containing the feedback operation enumeration value and the new correction value parameter. Extract the traceability index parameters from the confirmation record to retrieve and locate the multimodal event cluster and its corresponding clinical element, and use the feedback operation enumeration value to update the confirmation status value of the clinical element in the constraint verification layer.

9. An AI glasses-based multilingual audiovisual consultation and guidance system, characterized in that, The AI ​​glasses audiovisual multilingual fusion diagnostic guidance method as described in any one of claims 1-8 includes: AI glasses are used to collect first-person view images and environmental audio signals; The first-view visual understanding module is used to generate structured visual event data based on the first-view image. The auditory understanding module is used to generate structured data of auditory events based on the environmental audio signals; A language recognition and medical translation module is used to generate multilingual semantic data based on the environmental audio signals; The medical terminology protection module is used to extract medically sensitive fields from the multilingual semantic data and generate a medically sensitive field protection entity. The cross-modal semantic alignment module is used to aggregate the structured data of visual events, structured data of auditory events, multilingual semantic data, and the medical sensitive field protection entities that meet the dynamic time window constraints to generate a multimodal event cluster. The evidence verification matrix module is used to input the multimodal event cluster into the two-layer evidence verification matrix to deduce clinical elements and output clinical elements or conflict control signals with high confidence acceptance marks. An evidence-based medicine intelligent agent adapter is used to acquire a set of evidence-based context data using clinical elements with high confidence acceptance tags; The doctor's operation guidance generation module is used to generate operation guidance actions by combining the conflict control signal and the evidence-based context data set; The glasses-side limited display module is used to send the operation guidance actions to the AI ​​glasses for display. The bilingual confirmation module is used to collect feedback interaction signals to generate confirmation records, which are then sent back to the evidence verification matrix module to update the confirmation status value of the clinical elements based on the confirmation records.

10. The AI ​​glasses audiovisual multilingual fusion consultation and guidance system according to claim 9, characterized in that, The glasses-side limited display module is specifically used to parse the priority parameters of the operation guidance action, perform serialized queuing and scheduling, extract the text prompt field of the first-ranked operation guidance action and send it to the near-eye display component of the AI ​​glasses for visual rendering, and at the same time push the associated secondary evidence details into the display queue of the auxiliary terminal device. The bilingual confirmation module is specifically used to capture the voice, button or gesture feedback interaction signals of the operator in response to the operation guidance actions through the AI ​​glasses terminal, parse the feedback interaction signals into confirmation records of acceptance, modification, ignore or inapplicability, and write the confirmation records in reverse into the evidence mutual verification matrix module to cover the confirmation status value of the corresponding clinical elements.

Images